[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:44 dist-test-slave-wmg0 krb5kdc[10634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:44 dist-test-slave-wmg0 krb5kdc[10634](info): set up 2 sockets
Sep 27 10:44:44 dist-test-slave-wmg0 krb5kdc[10634](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 10:44:45.483292 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:44:45 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:44:45.603087 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:39343
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:45.710777 10647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:45.711040 10647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:45.711079 10647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:45.715078 10647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:45.715127 10647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:45.715145 10647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:45.715161 10647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:45.715176 10647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:45.720353 10647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:45.721344 10647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:45.722353 10647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:45.730926 10653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:45.730933 10652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:45.730969 10655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:45.731108 10647 server_base.cc:1034] running on GCE node
I20260927 10:44:45.731681 10647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:45.732797 10647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:45.733984 10647 hybrid_clock.cc:648] HybridClock initialized: now 1790505885733961 us; error 37 us; skew 500 ppm
Sep 27 10:44:45 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505885, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:45.741566 10647 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:44:45.742547 10647 webserver.cc:460] Webserver started at http://127.10.96.254:38883/ using document root <none> and password file <none>
I20260927 10:44:45.743117 10647 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:45.743211 10647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:45.743445 10647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:45.745527 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "0f22417cc07648c58a1ebc4f715f1b10"
format_stamp: "Formatted at 2026-09-27 10:44:45 on dist-test-slave-wmg0"
server_key: "ec6f17483752cc8ff91cceee36a50334"
server_key_iv: "d0522d09a522e5ed0a9fbc707d378e84"
server_key_version: "encryptionkey@0"
I20260927 10:44:45.746109 10647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "0f22417cc07648c58a1ebc4f715f1b10"
format_stamp: "Formatted at 2026-09-27 10:44:45 on dist-test-slave-wmg0"
server_key: "ec6f17483752cc8ff91cceee36a50334"
server_key_iv: "d0522d09a522e5ed0a9fbc707d378e84"
server_key_version: "encryptionkey@0"
I20260927 10:44:45.749804 10647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:45.752274 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:45.753558 10647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:45.753705 10647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "0f22417cc07648c58a1ebc4f715f1b10"
format_stamp: "Formatted at 2026-09-27 10:44:45 on dist-test-slave-wmg0"
server_key: "ec6f17483752cc8ff91cceee36a50334"
server_key_iv: "d0522d09a522e5ed0a9fbc707d378e84"
server_key_version: "encryptionkey@0"
I20260927 10:44:45.753813 10647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:45.767678 10647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:45.770793 10647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:45.770995 10647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:45.778043 10647 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:39343
I20260927 10:44:45.778060 10714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:39343 every 8 connection(s)
I20260927 10:44:45.778976 10647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:45.781572 10715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:45.786753 10715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10: Bootstrap starting.
I20260927 10:44:45.789193 10715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:45.789422 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 10647
I20260927 10:44:45.789603 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:44:45.790096 10715 log.cc:824] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:45.791131 10627 external_mini_cluster.cc:1428] Setting key c6453d621d78e6a5d336e4c41c8f291e
I20260927 10:44:45.792294 10715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10: No bootstrap required, opened a new log
I20260927 10:44:45.796562 10715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f22417cc07648c58a1ebc4f715f1b10" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39343 } }
I20260927 10:44:45.796808 10715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:45.796887 10715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f22417cc07648c58a1ebc4f715f1b10, State: Initialized, Role: FOLLOWER
I20260927 10:44:45.797313 10715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f22417cc07648c58a1ebc4f715f1b10" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39343 } }
I20260927 10:44:45.797469 10715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:45.797575 10715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:45.797703 10715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:45.798555 10715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f22417cc07648c58a1ebc4f715f1b10" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39343 } }
I20260927 10:44:45.798924 10715 leader_election.cc:302] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f22417cc07648c58a1ebc4f715f1b10; no voters:
I20260927 10:44:45.799223 10715 leader_election.cc:288] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:45.799369 10720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:45.799584 10720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [term 1 LEADER]: Becoming Leader. State: Replica: 0f22417cc07648c58a1ebc4f715f1b10, State: Running, Role: LEADER
I20260927 10:44:45.799876 10720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f22417cc07648c58a1ebc4f715f1b10" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39343 } }
I20260927 10:44:45.800412 10715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:45.802065 10721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f22417cc07648c58a1ebc4f715f1b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f22417cc07648c58a1ebc4f715f1b10" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39343 } } }
I20260927 10:44:45.802203 10721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:45.803092 10722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f22417cc07648c58a1ebc4f715f1b10. Latest consensus state: current_term: 1 leader_uuid: "0f22417cc07648c58a1ebc4f715f1b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f22417cc07648c58a1ebc4f715f1b10" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39343 } } }
I20260927 10:44:45.803194 10722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10 [sys.catalog]: This master's current role is: LEADER
W20260927 10:44:45.804975 10734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:44:45.805060 10734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:44:45.805130 10735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:45.805877 10735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 10:44:45 dist-test-slave-wmg0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:45.812671 10735 catalog_manager.cc:1347] Generated new cluster ID: b51d7cba32b9411db2aefb6ead756c3b
I20260927 10:44:45.812727 10735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:45.818045 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:45.794525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.10.96.254:39343)
0927 10:44:45.794979 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:45.794997 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:45.796113 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 10:44:45.799110 (+ 2997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:45.799129 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:45.799557 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:45.799899 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:45.803139 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:45.804412 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:45.806123 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:45.806790 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:45.806946 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:45.811741 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:45.811746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:45.811758 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:45.811783 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:45.814472 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:45.815111 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:45.815117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:45.815122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:45.815250 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:45.815635 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:45.815638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:45.815640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:45.816098 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:45.816277 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:45.816915 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 10:44:45.817198 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260927 10:44:45.829092 10735 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:45.830173 10735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:45.834600 10735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f22417cc07648c58a1ebc4f715f1b10: Generated new TSK 0
I20260927 10:44:45.835273 10735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:44:45.865854 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39343
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:45.999960 10741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:46.000221 10741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:46.000264 10741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:46.004333 10741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:46.004392 10741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:46.004452 10741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:44:46.009477 10741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:46.010500 10741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:46.011494 10741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:46.020860 10747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:46.020869 10746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:46.020869 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:46.021163 10741 server_base.cc:1034] running on GCE node
I20260927 10:44:46.021605 10741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:46.022248 10741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:46.025516 10741 hybrid_clock.cc:648] HybridClock initialized: now 1790505886025480 us; error 43 us; skew 500 ppm
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:46.028235 10741 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:44:46.029048 10741 webserver.cc:460] Webserver started at http://127.10.96.193:40069/ using document root <none> and password file <none>
I20260927 10:44:46.029652 10741 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:46.029750 10741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:46.029973 10741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:46.032008 10741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "0cfd2ea285d248fb8406afc4364a9536"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "40d1c6d8965633a23c8d27f295bd3468"
server_key_iv: "cbbbc629aeb9a2de9bc64baac5a66bb6"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.032557 10741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "0cfd2ea285d248fb8406afc4364a9536"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "40d1c6d8965633a23c8d27f295bd3468"
server_key_iv: "cbbbc629aeb9a2de9bc64baac5a66bb6"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.036111 10741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:44:46.038401 10756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.039568 10741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:46.039710 10741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "0cfd2ea285d248fb8406afc4364a9536"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "40d1c6d8965633a23c8d27f295bd3468"
server_key_iv: "cbbbc629aeb9a2de9bc64baac5a66bb6"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.039820 10741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:46.054209 10741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:46.057288 10741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:46.057538 10741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:46.058147 10741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:46.058923 10741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:46.058990 10741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.059051 10741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:46.059103 10741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.067615 10741 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:43255
I20260927 10:44:46.067644 10869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:43255 every 8 connection(s)
I20260927 10:44:46.068440 10741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:46.072516 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 10741
I20260927 10:44:46.072595 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:46.072841 10627 external_mini_cluster.cc:1428] Setting key 6afbecf2bc7c198816a70dd8bf971e42
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:44:46.082357 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.070059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:52295 (local address 127.10.96.254:39343)
0927 10:44:46.070211 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:46.070215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:46.071201 (+ 986us) server_negotiation.cc:407] Connection header received
0927 10:44:46.071976 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:46.071978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:46.072025 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:46.072096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:46.073346 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.073946 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.075053 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.075490 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.075611 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.078648 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:46.078652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:46.078655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:46.078685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:46.080255 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.080785 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.080790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.080792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.080870 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.081233 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.081237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.081238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.081415 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:46.081562 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:46.082037 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 10:44:46.082184 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:44:46.083101 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.070457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39343 (local address 127.10.96.193:52295)
0927 10:44:46.071054 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:46.071092 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:46.071806 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:46.072269 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:46.072276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:46.072663 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:46.073166 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:46.073179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.074049 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.074052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:46.074930 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:46.074939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.076043 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.076049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:46.076283 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.076882 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:46.076905 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:46.078490 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:46.080400 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:46.080406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:46.080420 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:46.080679 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:46.080974 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:46.080978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:46.080980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:46.081113 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:46.081561 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:46.081568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:46.081807 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 10:44:46.081993 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:44:46.082292 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":103,"threads_started":1}
I20260927 10:44:46.084411 10870 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:44:46.084641 10870 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:46.085160 10870 heartbeater.cc:511] Master 127.10.96.254:39343 requested a full tablet report, sending...
I20260927 10:44:46.086679 10679 ts_manager.cc:194] Registered new tserver with Master: 0cfd2ea285d248fb8406afc4364a9536 (127.10.96.193:43255)
I20260927 10:44:46.087766 10679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:52295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:44:46.098526 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39343
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:46.208700 10875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:46.208940 10875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:46.208982 10875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:46.212929 10875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:46.212983 10875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:46.213039 10875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:44:46.218076 10875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:46.218962 10875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:46.219856 10875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:46.229216 10881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:46.229207 10883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:46.229192 10880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:46.229243 10875 server_base.cc:1034] running on GCE node
I20260927 10:44:46.229841 10875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:46.230494 10875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:46.231685 10875 hybrid_clock.cc:648] HybridClock initialized: now 1790505886231671 us; error 26 us; skew 500 ppm
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:46.234804 10875 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:44:46.235656 10875 webserver.cc:460] Webserver started at http://127.10.96.194:35333/ using document root <none> and password file <none>
I20260927 10:44:46.236210 10875 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:46.236299 10875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:46.236519 10875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:46.238565 10875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "a92ad1c0170e492a98708b86c3ea624a"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "5747460ddb8880571c7b40d6cfd6c730"
server_key_iv: "4472d0f89c297c1ac9a7ae3b08adfdf3"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.239109 10875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "a92ad1c0170e492a98708b86c3ea624a"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "5747460ddb8880571c7b40d6cfd6c730"
server_key_iv: "4472d0f89c297c1ac9a7ae3b08adfdf3"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.242640 10875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 10:44:46.245018 10890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.246254 10875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:44:46.246389 10875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a92ad1c0170e492a98708b86c3ea624a"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "5747460ddb8880571c7b40d6cfd6c730"
server_key_iv: "4472d0f89c297c1ac9a7ae3b08adfdf3"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.246493 10875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:46.265815 10875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:46.268745 10875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:46.268955 10875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:46.269621 10875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:46.270421 10875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:46.270467 10875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.270506 10875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:46.270522 10875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.279771 10875 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:36611
I20260927 10:44:46.280100 11003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:36611 every 8 connection(s)
I20260927 10:44:46.280769 10875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:46.284520 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 10875
I20260927 10:44:46.284639 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:46.284955 10627 external_mini_cluster.cc:1428] Setting key 7d6d6c27f1a2aa7d36516afce5fced1a
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:44:46.294618 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.282641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:49835 (local address 127.10.96.254:39343)
0927 10:44:46.282782 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:46.282786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:46.283649 (+ 863us) server_negotiation.cc:407] Connection header received
0927 10:44:46.284555 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:46.284559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:46.284626 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:46.284741 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:46.286047 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.286569 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.287786 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.288009 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.288119 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.290948 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:46.290952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:46.290957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:46.290986 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:46.292539 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.293049 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.293054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.293056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.293152 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.293500 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.293504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.293506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.293696 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:46.293803 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:46.294325 (+ 522us) server_negotiation.cc:299] Negotiation successful
0927 10:44:46.294448 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:44:46.295395 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.283038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39343 (local address 127.10.96.194:49835)
0927 10:44:46.283484 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:46.283529 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:46.284317 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:46.284899 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:46.284907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:46.285353 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:46.285900 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:46.285914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.286712 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.286715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:46.287662 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:46.287671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.288162 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.288167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:46.288341 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.288994 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:46.289010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:46.290776 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:46.292667 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:46.292673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:46.292688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:46.292942 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:46.293251 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:46.293254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:46.293256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:46.293388 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:46.293800 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:46.293807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:46.294087 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 10:44:46.294286 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 10:44:46.294592 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:44:46.296571 11004 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39343
I20260927 10:44:46.296829 11004 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:46.297335 11004 heartbeater.cc:511] Master 127.10.96.254:39343 requested a full tablet report, sending...
I20260927 10:44:46.298424 10679 ts_manager.cc:194] Registered new tserver with Master: a92ad1c0170e492a98708b86c3ea624a (127.10.96.194:36611)
I20260927 10:44:46.299216 10679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:49835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:44:46.312381 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39343
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:46.422804 11009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:46.423053 11009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:46.423097 11009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:46.427194 11009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:46.427248 11009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:46.427302 11009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:44:46.432350 11009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39343
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:46.433275 11009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:46.434286 11009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:46.443516 11014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:46.443624 11009 server_base.cc:1034] running on GCE node
W20260927 10:44:46.443511 11017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:46.443848 11015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:46.444208 11009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:46.444880 11009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:46.446079 11009 hybrid_clock.cc:648] HybridClock initialized: now 1790505886446046 us; error 47 us; skew 500 ppm
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:46.449179 11009 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:44:46.450047 11009 webserver.cc:460] Webserver started at http://127.10.96.195:36241/ using document root <none> and password file <none>
I20260927 10:44:46.450644 11009 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:46.450730 11009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:46.450922 11009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:46.452863 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "fb093072d0cc46fb87e8d4abcac1988b"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "fdc202086cc196fc3e79f599cb31e783"
server_key_iv: "85d5384e893038db1d5496b2ba3bcf07"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.453452 11009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "fb093072d0cc46fb87e8d4abcac1988b"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "fdc202086cc196fc3e79f599cb31e783"
server_key_iv: "85d5384e893038db1d5496b2ba3bcf07"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.457024 11009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:46.459403 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.460594 11009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:44:46.460721 11009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "fb093072d0cc46fb87e8d4abcac1988b"
format_stamp: "Formatted at 2026-09-27 10:44:46 on dist-test-slave-wmg0"
server_key: "fdc202086cc196fc3e79f599cb31e783"
server_key_iv: "85d5384e893038db1d5496b2ba3bcf07"
server_key_version: "encryptionkey@0"
I20260927 10:44:46.460829 11009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:46.496289 11009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:46.499712 11009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:46.499960 11009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:46.500557 11009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:46.501426 11009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:46.501518 11009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.501603 11009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:46.501652 11009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:46.510489 11009 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40349
I20260927 10:44:46.510519 11137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40349 every 8 connection(s)
I20260927 10:44:46.511348 11009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:46.518494 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11009
I20260927 10:44:46.518625 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:46.518883 10627 external_mini_cluster.cc:1428] Setting key d7e8282246ebbcd61453dfb3e11bcda9
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:46.524497 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.513108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:34765 (local address 127.10.96.254:39343)
0927 10:44:46.513277 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:46.513282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:46.514390 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 10:44:46.515226 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:46.515229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:46.515278 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:46.515352 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:46.516484 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.516996 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.518171 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.518390 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.518504 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.521453 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:46.521456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:46.521459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:46.521508 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:46.522782 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.523241 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.523244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.523246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.523330 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.523616 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.523618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.523619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.523796 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:46.523870 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:46.524292 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 10:44:46.524382 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:44:46.525288 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.513552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39343 (local address 127.10.96.195:34765)
0927 10:44:46.514245 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:46.514286 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:46.515016 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:46.515512 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:46.515519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:46.515864 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:46.516361 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:46.516373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.517134 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.517136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:46.518053 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:46.518061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.518556 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.518560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:46.518805 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.519407 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:46.519422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:46.521284 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:46.522879 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:46.522884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:46.522895 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:46.523135 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:46.523416 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:46.523418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:46.523419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:46.523539 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:46.523879 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:46.523883 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:46.524089 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 10:44:46.524235 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:44:46.524486 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":108,"threads_started":1}
I20260927 10:44:46.526381 11138 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39343
I20260927 10:44:46.526578 11138 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:46.526981 11138 heartbeater.cc:511] Master 127.10.96.254:39343 requested a full tablet report, sending...
I20260927 10:44:46.527910 10679 ts_manager.cc:194] Registered new tserver with Master: fb093072d0cc46fb87e8d4abcac1988b (127.10.96.195:40349)
I20260927 10:44:46.528460 10679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:34765
I20260927 10:44:46.533190 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:44:46.559443 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.549523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.10.96.193:43255)
0927 10:44:46.549895 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:46.549909 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:46.549949 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:44:46.549998 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:46.550003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:46.550194 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:46.550332 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:46.551059 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.551845 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.552657 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.553062 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.553155 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.555630 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:46.555638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:46.555650 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:46.555690 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:46.557405 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.558070 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.558080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.558083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.558193 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.558594 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.558602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.558603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.558848 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:46.559000 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:46.559167 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 10:44:46.559291 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
W20260927 10:44:46.560464 10784 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39072
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[10634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:44:46.584661 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:46.576877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.10.96.193:43255)
0927 10:44:46.577052 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:46.577055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:46.577137 (+ 82us) server_negotiation.cc:407] Connection header received
0927 10:44:46.577297 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:46.577301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:46.577347 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:46.577423 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:46.578112 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.578615 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.579396 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:46.579614 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:46.579688 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:46.582016 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:46.582019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:46.582021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:46.582041 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:46.583305 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.583662 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.583664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.583666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.583736 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:46.583981 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:46.583984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:46.583988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:46.584141 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:46.584257 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:46.584454 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 10:44:46.584544 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:44:46.585719 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 10741
I20260927 10:44:46.591547 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 10875
I20260927 10:44:46.597117 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11009
I20260927 10:44:46.602528 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 10647
2026-09-27T10:44:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2162 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[11152](info): set up 2 sockets
Sep 27 10:44:46 dist-test-slave-wmg0 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260927 10:44:47.642009 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:44:47 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:44:47.758471 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:40353
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:40353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:47.867260 11165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:47.867568 11165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:47.867640 11165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:47.871848 11165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:47.871959 11165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:47.872000 11165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:47.872053 11165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:47.872090 11165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:47.878008 11165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:40353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:40353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:47.879094 11165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:47.880131 11165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:47.888322 11171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:47.888334 11170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:47.888384 11165 server_base.cc:1034] running on GCE node
W20260927 10:44:47.888334 11173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:47.889050 11165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:47.890184 11165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:47.891419 11165 hybrid_clock.cc:648] HybridClock initialized: now 1790505887891395 us; error 46 us; skew 500 ppm
Sep 27 10:44:47 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505887, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:47.894353 11165 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:44:47.895391 11165 webserver.cc:460] Webserver started at http://127.10.96.254:38455/ using document root <none> and password file <none>
I20260927 10:44:47.895948 11165 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:47.896044 11165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:47.896291 11165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:47.898316 11165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "30a12dbefc2d4c57897dd7e309791ad0"
format_stamp: "Formatted at 2026-09-27 10:44:47 on dist-test-slave-wmg0"
server_key: "d42f9ba034754f34b01e7e3b2565fa8a"
server_key_iv: "8b9e61de970a5a4cdcb68c46d96ecef1"
server_key_version: "encryptionkey@0"
I20260927 10:44:47.898859 11165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "30a12dbefc2d4c57897dd7e309791ad0"
format_stamp: "Formatted at 2026-09-27 10:44:47 on dist-test-slave-wmg0"
server_key: "d42f9ba034754f34b01e7e3b2565fa8a"
server_key_iv: "8b9e61de970a5a4cdcb68c46d96ecef1"
server_key_version: "encryptionkey@0"
I20260927 10:44:47.902387 11165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:44:47.904757 11180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:47.905994 11165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:47.906131 11165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "30a12dbefc2d4c57897dd7e309791ad0"
format_stamp: "Formatted at 2026-09-27 10:44:47 on dist-test-slave-wmg0"
server_key: "d42f9ba034754f34b01e7e3b2565fa8a"
server_key_iv: "8b9e61de970a5a4cdcb68c46d96ecef1"
server_key_version: "encryptionkey@0"
I20260927 10:44:47.906242 11165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:47.928318 11165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:47.931310 11165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:47.931504 11165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:47.938265 11165 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:40353
I20260927 10:44:47.938288 11232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:40353 every 8 connection(s)
I20260927 10:44:47.939167 11165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:47.941956 11233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:47.944536 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11165
I20260927 10:44:47.944624 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:44:47.944892 10627 external_mini_cluster.cc:1428] Setting key fe05b18a1e5f651e9a3454110f4fd0a0
I20260927 10:44:47.947525 11233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0: Bootstrap starting.
I20260927 10:44:47.950065 11233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:47.950950 11233 log.cc:824] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:47.952895 11233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0: No bootstrap required, opened a new log
Sep 27 10:44:47 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:47.956550 11233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a12dbefc2d4c57897dd7e309791ad0" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40353 } }
I20260927 10:44:47.956848 11233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:47.956940 11233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a12dbefc2d4c57897dd7e309791ad0, State: Initialized, Role: FOLLOWER
I20260927 10:44:47.957420 11233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a12dbefc2d4c57897dd7e309791ad0" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40353 } }
I20260927 10:44:47.957595 11233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:47.957674 11233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:47.957813 11233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:47.958684 11233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a12dbefc2d4c57897dd7e309791ad0" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40353 } }
I20260927 10:44:47.959072 11233 leader_election.cc:302] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30a12dbefc2d4c57897dd7e309791ad0; no voters:
I20260927 10:44:47.959415 11233 leader_election.cc:288] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:47.959499 11238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:47.959758 11238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 30a12dbefc2d4c57897dd7e309791ad0, State: Running, Role: LEADER
I20260927 10:44:47.960088 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:47.946284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.10.96.254:40353)
0927 10:44:47.946676 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:47.946700 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:47.946734 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:44:47.947329 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:47.947346 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:47.947694 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:47.947972 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:47.948824 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:47.949912 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:47.951112 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:47.951612 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:47.951801 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:47.954522 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:47.954527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:47.954540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:47.954566 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:47.956932 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:47.957448 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:47.957471 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:47.957478 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:47.957634 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:47.957922 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:47.957925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:47.957927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:47.958324 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:47.958532 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:47.958812 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 10:44:47.959100 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":132,"threads_started":1}
I20260927 10:44:47.960107 11238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a12dbefc2d4c57897dd7e309791ad0" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40353 } }
I20260927 10:44:47.960731 11233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:47.962037 11240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30a12dbefc2d4c57897dd7e309791ad0. Latest consensus state: current_term: 1 leader_uuid: "30a12dbefc2d4c57897dd7e309791ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a12dbefc2d4c57897dd7e309791ad0" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40353 } } }
I20260927 10:44:47.962152 11240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:47.962579 11239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30a12dbefc2d4c57897dd7e309791ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30a12dbefc2d4c57897dd7e309791ad0" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40353 } } }
I20260927 10:44:47.962662 11239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0 [sys.catalog]: This master's current role is: LEADER
W20260927 10:44:47.965968 11253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:44:47.966048 11253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:44:47.966104 11247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:47.966928 11247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:47.973044 11247 catalog_manager.cc:1347] Generated new cluster ID: fcdac45beb0c4c7f83c653471b8be6ef
I20260927 10:44:47.973134 11247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:47.992048 11247 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:47.993024 11247 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:48.000468 11247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30a12dbefc2d4c57897dd7e309791ad0: Generated new TSK 0
I20260927 10:44:48.001375 11247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:44:48.034173 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40353
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:48.141525 11259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:48.141772 11259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:48.141814 11259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:48.145802 11259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:48.145856 11259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:48.145949 11259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:44:48.151031 11259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40353
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:48.152026 11259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:48.153033 11259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:48.162808 11264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:48.162815 11265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:48.162819 11267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:48.162854 11259 server_base.cc:1034] running on GCE node
I20260927 10:44:48.163383 11259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:48.163996 11259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:48.165169 11259 hybrid_clock.cc:648] HybridClock initialized: now 1790505888165140 us; error 50 us; skew 500 ppm
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:48.167766 11259 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:44:48.168613 11259 webserver.cc:460] Webserver started at http://127.10.96.193:42239/ using document root <none> and password file <none>
I20260927 10:44:48.169142 11259 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:48.169227 11259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:48.169519 11259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:48.171475 11259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "34188a833b3c49dfb59e61214bd304e0"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "9c0fd355e18532820484ba6b54a42d52"
server_key_iv: "0e4d40091b25c665b5a9be1b7025c50e"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.172013 11259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "34188a833b3c49dfb59e61214bd304e0"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "9c0fd355e18532820484ba6b54a42d52"
server_key_iv: "0e4d40091b25c665b5a9be1b7025c50e"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.175465 11259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:44:48.177716 11274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.178805 11259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:44:48.178927 11259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "34188a833b3c49dfb59e61214bd304e0"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "9c0fd355e18532820484ba6b54a42d52"
server_key_iv: "0e4d40091b25c665b5a9be1b7025c50e"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.179025 11259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:48.190619 11259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:48.193145 11259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:48.193329 11259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:48.193889 11259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:48.194624 11259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:48.194697 11259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.194787 11259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:48.194832 11259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.203287 11259 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:46553
I20260927 10:44:48.203322 11387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:46553 every 8 connection(s)
I20260927 10:44:48.204095 11259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:48.209975 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11259
I20260927 10:44:48.210100 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:48.210337 10627 external_mini_cluster.cc:1428] Setting key b625f97fcbaf18a82eae90417e8e0778
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:48.218186 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.205811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55657 (local address 127.10.96.254:40353)
0927 10:44:48.205928 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.205931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.206679 (+ 748us) server_negotiation.cc:407] Connection header received
0927 10:44:48.207898 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.207901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.207949 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.208023 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.209138 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.209779 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.211167 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.211489 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.211607 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.214228 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.214232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.214234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.214257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.215987 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.216481 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.216488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.216491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.216595 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.216900 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.216905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.216909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.217192 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.217296 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.217804 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.217939 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:44:48.218792 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.206128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:40353 (local address 127.10.96.193:55657)
0927 10:44:48.206529 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.206568 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.207703 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.208151 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.208157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.208483 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:48.208970 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.209010 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.209906 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.209908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.211061 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.211070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.211641 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.211646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.211827 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.212411 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:48.212426 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:48.214040 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:48.216103 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:48.216107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:48.216118 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:48.216384 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:48.216687 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:48.216690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:48.216692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:48.216812 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:48.217292 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:48.217296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:48.217571 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.217725 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.217955 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:44:48.220001 11388 heartbeater.cc:347] Connected to a master server at 127.10.96.254:40353
I20260927 10:44:48.220260 11388 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:48.220808 11388 heartbeater.cc:511] Master 127.10.96.254:40353 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:44:48.222405 11197 ts_manager.cc:194] Registered new tserver with Master: 34188a833b3c49dfb59e61214bd304e0 (127.10.96.193:46553)
I20260927 10:44:48.223888 11197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:55657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:44:48.236292 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40353
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:48.344043 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:48.344282 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:48.344354 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:48.348371 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:48.348418 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:48.348503 11393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:44:48.353528 11393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40353
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:48.354489 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:48.355455 11393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:48.364974 11398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:48.364974 11399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:48.365127 11393 server_base.cc:1034] running on GCE node
W20260927 10:44:48.364951 11401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:48.365612 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:48.366197 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:48.367367 11393 hybrid_clock.cc:648] HybridClock initialized: now 1790505888367353 us; error 25 us; skew 500 ppm
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:48.370002 11393 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:44:48.370836 11393 webserver.cc:460] Webserver started at http://127.10.96.194:44259/ using document root <none> and password file <none>
I20260927 10:44:48.371371 11393 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:48.371454 11393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:48.371663 11393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:48.373684 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "5c6d5067dd274d73a92c649a8a0db4aa"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "984b288704259b64ce7f2c5159ca64b5"
server_key_iv: "a1404bbf97ea5bff32c0b31eb455ff88"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.374221 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "5c6d5067dd274d73a92c649a8a0db4aa"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "984b288704259b64ce7f2c5159ca64b5"
server_key_iv: "a1404bbf97ea5bff32c0b31eb455ff88"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.377677 11393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:48.379904 11408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.381057 11393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:48.381182 11393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "5c6d5067dd274d73a92c649a8a0db4aa"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "984b288704259b64ce7f2c5159ca64b5"
server_key_iv: "a1404bbf97ea5bff32c0b31eb455ff88"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.381294 11393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:48.404677 11393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:48.407626 11393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:48.407826 11393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:48.408368 11393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:48.409190 11393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:48.409252 11393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.409317 11393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:48.409356 11393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.417903 11393 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:41339
I20260927 10:44:48.417928 11521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:41339 every 8 connection(s)
I20260927 10:44:48.418725 11393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:48.421943 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11393
I20260927 10:44:48.422065 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:48.422341 10627 external_mini_cluster.cc:1428] Setting key b26102ad2e0fb14ee455067b73e04e9f
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:44:48.432914 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.420362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:41079 (local address 127.10.96.254:40353)
0927 10:44:48.420482 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.420485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.421298 (+ 813us) server_negotiation.cc:407] Connection header received
0927 10:44:48.422682 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.422685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.422742 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.422843 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.424149 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.424666 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.426416 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.426640 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.426744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.429452 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.429455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.429492 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.429527 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.430986 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.431562 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.431565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.431566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.431650 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.431932 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.431934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.431935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.432140 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.432249 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.432669 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.432770 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:44:48.433873 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.420737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:40353 (local address 127.10.96.194:41079)
0927 10:44:48.421159 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.421198 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.422476 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.423144 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.423151 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.423569 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:48.424021 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.424034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.424779 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.424782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.426286 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.426296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.426773 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.426777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.426954 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.427507 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:48.427523 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:48.429283 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:48.431102 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:48.431112 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:48.431128 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:48.431470 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:48.431732 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:48.431734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:48.431736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:48.431857 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:48.432247 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:48.432251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:48.432467 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.432618 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.432859 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260927 10:44:48.435043 11522 heartbeater.cc:347] Connected to a master server at 127.10.96.254:40353
I20260927 10:44:48.435316 11522 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:48.435797 11522 heartbeater.cc:511] Master 127.10.96.254:40353 requested a full tablet report, sending...
I20260927 10:44:48.436851 11197 ts_manager.cc:194] Registered new tserver with Master: 5c6d5067dd274d73a92c649a8a0db4aa (127.10.96.194:41339)
I20260927 10:44:48.437426 11197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:41079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:44:48.448863 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40353
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:48.556074 11527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:48.556349 11527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:48.556421 11527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:48.560753 11527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:48.560818 11527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:48.560899 11527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:44:48.566063 11527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40353
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:48.567025 11527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:48.567979 11527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:48.576943 11533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:48.577077 11527 server_base.cc:1034] running on GCE node
W20260927 10:44:48.576967 11532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:48.576943 11535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:48.577658 11527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:48.578264 11527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:48.579437 11527 hybrid_clock.cc:648] HybridClock initialized: now 1790505888579427 us; error 30 us; skew 500 ppm
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:48.581990 11527 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:44:48.582870 11527 webserver.cc:460] Webserver started at http://127.10.96.195:42893/ using document root <none> and password file <none>
I20260927 10:44:48.583433 11527 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:48.583524 11527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:48.583736 11527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:48.585789 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "d4ee406e084e47248f65360d726ef946"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "6d96c03b57d81fad85a9a64ef3a4840a"
server_key_iv: "649c7cf5af793e4f8dc2a69bfe174a97"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.586323 11527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "d4ee406e084e47248f65360d726ef946"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "6d96c03b57d81fad85a9a64ef3a4840a"
server_key_iv: "649c7cf5af793e4f8dc2a69bfe174a97"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.589820 11527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:48.592021 11542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.593164 11527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:48.593330 11527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "d4ee406e084e47248f65360d726ef946"
format_stamp: "Formatted at 2026-09-27 10:44:48 on dist-test-slave-wmg0"
server_key: "6d96c03b57d81fad85a9a64ef3a4840a"
server_key_iv: "649c7cf5af793e4f8dc2a69bfe174a97"
server_key_version: "encryptionkey@0"
I20260927 10:44:48.593443 11527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:48.634495 11527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:48.637436 11527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:48.637652 11527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:48.638238 11527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:48.639029 11527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:48.639093 11527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.639158 11527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:48.639209 11527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:48.647444 11527 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:44659
I20260927 10:44:48.647475 11655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:44659 every 8 connection(s)
I20260927 10:44:48.648299 11527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:48.654515 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11527
I20260927 10:44:48.654630 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:48.654925 10627 external_mini_cluster.cc:1428] Setting key 47bcea117df23587af838c64d98eae20
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:48.661935 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.649991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56017 (local address 127.10.96.254:40353)
0927 10:44:48.650119 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.650123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.650942 (+ 819us) server_negotiation.cc:407] Connection header received
0927 10:44:48.652286 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.652291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.652391 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.652521 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.653689 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.654505 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.656017 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.656285 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.656390 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.658835 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.658838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.658840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.658863 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.660104 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.660578 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.660580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.660581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.660651 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.660977 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.660979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.660980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.661157 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.661228 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.661685 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.661791 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:44:48.662804 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.650376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:40353 (local address 127.10.96.195:56017)
0927 10:44:48.650805 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.650845 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.652009 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.652698 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.652705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.653050 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:48.653557 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.653572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.654664 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.654671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.655885 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.655894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.656423 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.656426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.656558 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.657165 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:48.657180 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:48.658692 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:48.660225 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:48.660231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:48.660242 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:48.660502 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:48.660773 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:48.660775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:48.660776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:48.660901 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:48.661236 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:48.661241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:48.661449 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.661651 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.661994 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260927 10:44:48.663966 11656 heartbeater.cc:347] Connected to a master server at 127.10.96.254:40353
I20260927 10:44:48.664153 11656 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:48.664552 11656 heartbeater.cc:511] Master 127.10.96.254:40353 requested a full tablet report, sending...
I20260927 10:44:48.665437 11197 ts_manager.cc:194] Registered new tserver with Master: d4ee406e084e47248f65360d726ef946 (127.10.96.195:44659)
I20260927 10:44:48.666021 11197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:56017
I20260927 10:44:48.668383 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:44:48.678236 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.671511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.10.96.254:40353)
0927 10:44:48.671619 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.671622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.671919 (+ 297us) server_negotiation.cc:407] Connection header received
0927 10:44:48.672149 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.672152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.672213 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.672282 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.672889 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.673450 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.674264 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.674466 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.674538 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.675665 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.675668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.675670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.675691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.676875 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.677258 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.677261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.677263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.677355 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.677583 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.677586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.677587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.677765 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.677863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.678027 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.678108 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:44:48.682498 11197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:48.685629 11197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:48.699363 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.694251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.10.96.193:46553)
0927 10:44:48.694621 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.694634 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.694865 (+ 231us) server_negotiation.cc:407] Connection header received
0927 10:44:48.694969 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.694975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.695139 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.695271 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:48.695647 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.696732 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.698093 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.698712 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.698843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.698982 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.699120 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.699213 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":60,"threads_started":1}
I20260927 10:44:48.699368 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.693375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:41339 (local address 127.0.0.1:44514)
0927 10:44:48.693808 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.693850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.693971 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.695071 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.695077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.695099 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:48.695258 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.695265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.696448 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.696451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.697869 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.697879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.698942 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.698946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.699044 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.699069 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.699117 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.699219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":98,"threads_started":1}
I20260927 10:44:48.699378 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.693718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:46553 (local address 127.0.0.1:53092)
0927 10:44:48.694745 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.694758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.694865 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.695429 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.695432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.695450 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:48.695527 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.695534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.696909 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.696911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.697958 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.697966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.698840 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.698843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.698951 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.698989 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.699094 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.699242 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260927 10:44:48.699733 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.693451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:44659 (local address 127.0.0.1:41522)
0927 10:44:48.694451 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.694464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.694550 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.695214 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.695217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.695232 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:48.695356 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.695365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.696867 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.696871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.698175 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.698184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.699339 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.699342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.699437 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.699464 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.699534 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.699598 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260927 10:44:48.700125 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.694052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.10.96.194:41339)
0927 10:44:48.694482 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.694497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.694520 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:44:48.694597 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.694603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.694774 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.695398 (+ 624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:48.695549 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.696305 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.698002 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.698802 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.699716 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.699858 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.699909 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.699995 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":131,"threads_started":1}
I20260927 10:44:48.700196 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.694156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.10.96.195:44659)
0927 10:44:48.694588 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.694600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.694622 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:44:48.694701 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.694708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.694909 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.695055 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:48.695484 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.696746 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.698297 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.699060 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.699166 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.699283 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.699975 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.700053 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":152,"threads_started":1}
I20260927 10:44:48.702529 11590 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0f8d6822ef44eb8e3780a51eb11249 (DEFAULT_TABLE table=test-table [id=a2c06b5cd62b4695a6b93ec1f35ceca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:48.702528 11322 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0f8d6822ef44eb8e3780a51eb11249 (DEFAULT_TABLE table=test-table [id=a2c06b5cd62b4695a6b93ec1f35ceca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:48.703418 11322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0f8d6822ef44eb8e3780a51eb11249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:48.703426 11590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0f8d6822ef44eb8e3780a51eb11249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:48.708994 11673 tablet_bootstrap.cc:492] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: Bootstrap starting.
I20260927 10:44:48.710197 11674 tablet_bootstrap.cc:492] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0: Bootstrap starting.
I20260927 10:44:48.711464 11673 tablet_bootstrap.cc:654] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:48.711356 11456 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0f8d6822ef44eb8e3780a51eb11249 (DEFAULT_TABLE table=test-table [id=a2c06b5cd62b4695a6b93ec1f35ceca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:48.712002 11674 tablet_bootstrap.cc:654] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:48.712448 11456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0f8d6822ef44eb8e3780a51eb11249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:48.712702 11673 log.cc:824] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:48.713006 11674 log.cc:824] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:48.714669 11673 tablet_bootstrap.cc:492] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: No bootstrap required, opened a new log
I20260927 10:44:48.714854 11673 ts_tablet_manager.cc:1397] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:44:48.714879 11674 tablet_bootstrap.cc:492] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0: No bootstrap required, opened a new log
I20260927 10:44:48.715021 11674 ts_tablet_manager.cc:1397] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:44:48.718551 11677 tablet_bootstrap.cc:492] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: Bootstrap starting.
I20260927 10:44:48.719077 11673 raft_consensus.cc:348] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.719077 11674 raft_consensus.cc:348] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.719342 11673 raft_consensus.cc:374] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:48.719342 11674 raft_consensus.cc:374] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:48.719400 11674 raft_consensus.cc:729] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34188a833b3c49dfb59e61214bd304e0, State: Initialized, Role: FOLLOWER
I20260927 10:44:48.719431 11673 raft_consensus.cc:729] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ee406e084e47248f65360d726ef946, State: Initialized, Role: FOLLOWER
I20260927 10:44:48.719892 11674 consensus_queue.cc:260] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.719892 11673 consensus_queue.cc:260] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.720480 11388 heartbeater.cc:503] Master 127.10.96.254:40353 was elected leader, sending a full tablet report...
I20260927 10:44:48.720835 11673 ts_tablet_manager.cc:1428] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:44:48.720835 11674 ts_tablet_manager.cc:1428] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:44:48.721156 11656 heartbeater.cc:503] Master 127.10.96.254:40353 was elected leader, sending a full tablet report...
I20260927 10:44:48.721238 11677 tablet_bootstrap.cc:654] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:48.722126 11677 log.cc:824] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:48.724218 11677 tablet_bootstrap.cc:492] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: No bootstrap required, opened a new log
I20260927 10:44:48.724422 11677 ts_tablet_manager.cc:1397] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260927 10:44:48.727245 11677 raft_consensus.cc:348] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.727466 11677 raft_consensus.cc:374] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:48.727530 11677 raft_consensus.cc:729] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d5067dd274d73a92c649a8a0db4aa, State: Initialized, Role: FOLLOWER
I20260927 10:44:48.727912 11677 consensus_queue.cc:260] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.728470 11522 heartbeater.cc:503] Master 127.10.96.254:40353 was elected leader, sending a full tablet report...
I20260927 10:44:48.728797 11677 ts_tablet_manager.cc:1428] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20260927 10:44:48.847440 11679 raft_consensus.cc:482] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:48.847687 11679 raft_consensus.cc:504] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.848826 11679 leader_election.cc:288] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c6d5067dd274d73a92c649a8a0db4aa (127.10.96.194:41339), 34188a833b3c49dfb59e61214bd304e0 (127.10.96.193:46553)
I20260927 10:44:48.852188 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.849174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:50263 (local address 127.10.96.193:46553)
0927 10:44:48.849302 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.849307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.849335 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:44:48.849394 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.849397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.849434 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.849544 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:48.849794 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.850295 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.851298 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.851669 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.851795 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.851845 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.852010 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.852073 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:44:48.852198 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.849045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:46553 (local address 127.10.96.195:50263)
0927 10:44:48.849155 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.849170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.849284 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.849550 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.849554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.849570 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:48.849683 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.849690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.850398 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.850404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.851172 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.851180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.851786 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.851790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.851896 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.851914 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.851996 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.852063 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 10:44:48.852555 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.849155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:36179 (local address 127.10.96.194:41339)
0927 10:44:48.849308 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.849314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.849605 (+ 291us) server_negotiation.cc:407] Connection header received
0927 10:44:48.849746 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.849749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.849795 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.849878 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:48.850110 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.850580 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.851582 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.851984 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.852108 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.852154 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.852371 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.852445 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:48.852859 11682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.849050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:41339 (local address 127.10.96.195:36179)
0927 10:44:48.849510 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:48.849521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:48.849638 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:48.849883 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:48.849886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:48.849900 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:48.850006 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.850012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.850698 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.850702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.851464 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:48.851472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.852098 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.852101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:48.852203 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.852227 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:44:48.852697 (+ 470us) client_negotiation.cc:240] Negotiation successful
0927 10:44:48.852764 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":104,"thread_start_us":132,"threads_started":1}
I20260927 10:44:48.852838 11342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0f8d6822ef44eb8e3780a51eb11249" candidate_uuid: "d4ee406e084e47248f65360d726ef946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34188a833b3c49dfb59e61214bd304e0" is_pre_election: true
I20260927 10:44:48.853159 11342 raft_consensus.cc:2454] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ee406e084e47248f65360d726ef946 in term 0.
I20260927 10:44:48.853475 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0f8d6822ef44eb8e3780a51eb11249" candidate_uuid: "d4ee406e084e47248f65360d726ef946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" is_pre_election: true
I20260927 10:44:48.853678 11546 leader_election.cc:302] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34188a833b3c49dfb59e61214bd304e0, d4ee406e084e47248f65360d726ef946; no voters:
I20260927 10:44:48.853744 11476 raft_consensus.cc:2454] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4ee406e084e47248f65360d726ef946 in term 0.
I20260927 10:44:48.853937 11679 raft_consensus.cc:2781] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:48.854035 11679 raft_consensus.cc:482] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:48.854108 11679 raft_consensus.cc:3037] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:48.855111 11679 raft_consensus.cc:504] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.855433 11679 leader_election.cc:288] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [CANDIDATE]: Term 1 election: Requested vote from peers 5c6d5067dd274d73a92c649a8a0db4aa (127.10.96.194:41339), 34188a833b3c49dfb59e61214bd304e0 (127.10.96.193:46553)
I20260927 10:44:48.855731 11476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0f8d6822ef44eb8e3780a51eb11249" candidate_uuid: "d4ee406e084e47248f65360d726ef946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c6d5067dd274d73a92c649a8a0db4aa"
I20260927 10:44:48.855849 11476 raft_consensus.cc:3037] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:48.855872 11342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0f8d6822ef44eb8e3780a51eb11249" candidate_uuid: "d4ee406e084e47248f65360d726ef946" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34188a833b3c49dfb59e61214bd304e0"
I20260927 10:44:48.855998 11342 raft_consensus.cc:3037] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:48.856767 11476 raft_consensus.cc:2454] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ee406e084e47248f65360d726ef946 in term 1.
I20260927 10:44:48.857038 11544 leader_election.cc:302] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c6d5067dd274d73a92c649a8a0db4aa, d4ee406e084e47248f65360d726ef946; no voters:
I20260927 10:44:48.857064 11342 raft_consensus.cc:2454] T 8d0f8d6822ef44eb8e3780a51eb11249 P 34188a833b3c49dfb59e61214bd304e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4ee406e084e47248f65360d726ef946 in term 1.
I20260927 10:44:48.857229 11679 raft_consensus.cc:2781] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:48.857491 11679 raft_consensus.cc:686] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [term 1 LEADER]: Becoming Leader. State: Replica: d4ee406e084e47248f65360d726ef946, State: Running, Role: LEADER
I20260927 10:44:48.857811 11679 consensus_queue.cc:237] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } }
I20260927 10:44:48.860725 11197 catalog_manager.cc:5495] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4ee406e084e47248f65360d726ef946 (127.10.96.195). New cstate: current_term: 1 leader_uuid: "d4ee406e084e47248f65360d726ef946" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c6d5067dd274d73a92c649a8a0db4aa" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4ee406e084e47248f65360d726ef946" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34188a833b3c49dfb59e61214bd304e0" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 46553 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:44:48.889153 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.880564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.10.96.193:46553)
0927 10:44:48.880748 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.880752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.880814 (+ 62us) server_negotiation.cc:407] Connection header received
0927 10:44:48.880926 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.880928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.880968 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.881037 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.881790 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.882314 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.883195 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.883403 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.883481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.886135 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.886146 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.886162 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.886187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.887720 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.888103 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.888109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.888110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.888194 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.888417 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.888419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.888420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.888647 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.888822 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.888942 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.889036 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 10:44:48.899192 11657 tablet.cc:2369] T 8d0f8d6822ef44eb8e3780a51eb11249 P d4ee406e084e47248f65360d726ef946: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:44:48.914750 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.906870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.10.96.193:46553)
0927 10:44:48.907037 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.907047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.907110 (+ 63us) server_negotiation.cc:407] Connection header received
0927 10:44:48.907256 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.907259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.907300 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.907365 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.907979 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.908486 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.909350 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.909630 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.909707 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.912067 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.912070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.912072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.912094 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.913340 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.913761 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.913764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.913765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.913832 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.914033 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.914034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.914035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.914166 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.914280 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.914546 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.914638 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 10:44:48.919797 11523 tablet.cc:2369] T 8d0f8d6822ef44eb8e3780a51eb11249 P 5c6d5067dd274d73a92c649a8a0db4aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:44:48 dist-test-slave-wmg0 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:44:48.939597 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:48.931219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.10.96.193:46553)
0927 10:44:48.931385 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:48.931395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:48.931550 (+ 155us) server_negotiation.cc:407] Connection header received
0927 10:44:48.931724 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:48.931727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:48.931769 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:48.931849 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:48.932551 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.933053 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.933931 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:48.934144 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:48.934248 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:48.936599 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:48.936602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:48.936604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:48.936627 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:48.938055 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.938534 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.938537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.938538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.938606 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:48.938877 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:48.938880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:48.938881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:48.939031 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:48.939123 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:48.939373 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 10:44:48.939462 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:44:48.941037 11302 tablet_service.cc:2908] Rejecting scan request for tablet 8d0f8d6822ef44eb8e3780a51eb11249: Uninitialized: clean time has not yet been initialized
I20260927 10:44:48.943173 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11259
I20260927 10:44:48.949222 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11393
I20260927 10:44:48.954842 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11527
I20260927 10:44:48.960104 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11165
2026-09-27T10:44:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2356 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:49 dist-test-slave-wmg0 krb5kdc[11697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:49 dist-test-slave-wmg0 krb5kdc[11697](info): set up 2 sockets
Sep 27 10:44:49 dist-test-slave-wmg0 krb5kdc[11697](info): commencing operation
krb5kdc: starting...
W20260927 10:44:49.989982 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:44:50.107643 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43335
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:50.215644 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:50.215914 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:50.215955 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:50.220057 11710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:50.220118 11710 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:44:50.220136 11710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:50.220155 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:50.220170 11710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:50.220186 11710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:50.225667 11710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43335
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:50.226682 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:50.227695 11710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:50.236374 11710 server_base.cc:1034] running on GCE node
W20260927 10:44:50.236322 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.236318 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.236325 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:50.237144 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:50.238317 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:50.239500 11710 hybrid_clock.cc:648] HybridClock initialized: now 1790505890239479 us; error 36 us; skew 500 ppm
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:50.242406 11710 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:44:50.243319 11710 webserver.cc:460] Webserver started at http://127.10.96.254:44795/ using document root <none> and password file <none>
I20260927 10:44:50.243918 11710 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:50.244005 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:50.244246 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:50.246385 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "238462aeef994682acf920932517873f"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "5bd0d3f64466fccf93bf6a95e834476e"
server_key_iv: "2244c207d88e301bd03e7e99e6c48b6f"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.246968 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "238462aeef994682acf920932517873f"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "5bd0d3f64466fccf93bf6a95e834476e"
server_key_iv: "2244c207d88e301bd03e7e99e6c48b6f"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.250777 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:44:50.253305 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.254616 11710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:50.254758 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "238462aeef994682acf920932517873f"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "5bd0d3f64466fccf93bf6a95e834476e"
server_key_iv: "2244c207d88e301bd03e7e99e6c48b6f"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.254866 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:50.265690 11710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:50.268401 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:50.268592 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:50.276051 11710 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43335
I20260927 10:44:50.276118 11787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43335 every 8 connection(s)
I20260927 10:44:50.276968 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:50.279670 11788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:50.283753 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11710
I20260927 10:44:50.283876 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:44:50.284121 10627 external_mini_cluster.cc:1428] Setting key 71faf9dc6e4cd6e5b99540bfc21e6d44
I20260927 10:44:50.285681 11788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: Bootstrap starting.
I20260927 10:44:50.287842 11788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:50.288563 11788 log.cc:824] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:50.290398 11788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: No bootstrap required, opened a new log
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:50.294242 11788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238462aeef994682acf920932517873f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43335 } }
I20260927 10:44:50.294516 11788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:50.294588 11788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238462aeef994682acf920932517873f, State: Initialized, Role: FOLLOWER
I20260927 10:44:50.295015 11788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238462aeef994682acf920932517873f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43335 } }
I20260927 10:44:50.295135 11788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:50.295220 11788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:50.295342 11788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:50.296185 11788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238462aeef994682acf920932517873f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43335 } }
I20260927 10:44:50.296571 11788 leader_election.cc:302] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 238462aeef994682acf920932517873f; no voters:
I20260927 10:44:50.296892 11788 leader_election.cc:288] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:50.296993 11793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:50.297232 11793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [term 1 LEADER]: Becoming Leader. State: Replica: 238462aeef994682acf920932517873f, State: Running, Role: LEADER
I20260927 10:44:50.297567 11793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238462aeef994682acf920932517873f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43335 } }
I20260927 10:44:50.298167 11788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:50.299134 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.285518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.10.96.254:43335)
0927 10:44:50.286026 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:50.286041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:50.286076 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:44:50.286704 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:50.286721 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:50.287034 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:50.287366 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:50.288157 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.289173 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.290058 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.290567 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.290739 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.293337 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:50.293342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:50.293355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:50.293379 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:50.295572 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.295996 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.296002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.296007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.296120 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.296434 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.296438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.296439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.296976 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:50.297186 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:50.297837 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 10:44:50.298133 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":145,"threads_started":1}
I20260927 10:44:50.299108 11795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "238462aeef994682acf920932517873f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238462aeef994682acf920932517873f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43335 } } }
I20260927 10:44:50.299216 11795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:50.299417 11795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 238462aeef994682acf920932517873f. Latest consensus state: current_term: 1 leader_uuid: "238462aeef994682acf920932517873f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238462aeef994682acf920932517873f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43335 } } }
I20260927 10:44:50.299502 11795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:50.299769 11802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:50.302838 11802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:50.308539 11802 catalog_manager.cc:1347] Generated new cluster ID: 935132f5e7a9471aabebcf73c6536fed
I20260927 10:44:50.308612 11802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:50.315274 11802 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:50.316164 11802 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:50.323467 11802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: Generated new TSK 0
I20260927 10:44:50.324669 11802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:44:50.357645 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43335
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:50.465662 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:50.465921 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:50.465996 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:50.470082 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:50.470139 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:50.470182 11814 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:50.470216 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:44:50.475505 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:50.476553 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:50.477607 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:50.486837 11820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.486866 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:50.486919 11814 server_base.cc:1034] running on GCE node
W20260927 10:44:50.486866 11819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:50.487386 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:50.487996 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:50.489176 11814 hybrid_clock.cc:648] HybridClock initialized: now 1790505890489157 us; error 31 us; skew 500 ppm
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:50.497203 11814 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:44:50.498106 11814 webserver.cc:460] Webserver started at http://127.10.96.193:33501/ using document root <none> and password file <none>
I20260927 10:44:50.498667 11814 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:50.498769 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:50.498991 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:50.502393 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "4f27f86c137640608976876a64f6d39e"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "bc89538bf07cd61af299330cc7583cbc"
server_key_iv: "fa6e9e15e9b3a8e2ce755740a4a2bd7e"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.503165 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "4f27f86c137640608976876a64f6d39e"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "bc89538bf07cd61af299330cc7583cbc"
server_key_iv: "fa6e9e15e9b3a8e2ce755740a4a2bd7e"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.509085 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 10:44:50.512504 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.514219 11814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:50.514350 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "4f27f86c137640608976876a64f6d39e"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "bc89538bf07cd61af299330cc7583cbc"
server_key_iv: "fa6e9e15e9b3a8e2ce755740a4a2bd7e"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.514609 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:50.531674 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:50.535573 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:50.535765 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:50.537317 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:50.537390 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.537454 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:50.537513 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:50.549588 11814 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39329
I20260927 10:44:50.549826 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39329 every 8 connection(s)
I20260927 10:44:50.550510 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:50.553748 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11814
I20260927 10:44:50.553867 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:50.554201 10627 external_mini_cluster.cc:1428] Setting key 96a379a1da56fc30d8b31926ed721696
I20260927 10:44:50.554793 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.539950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:49253 (local address 127.10.96.254:43335)
0927 10:44:50.540059 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:50.540064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:50.540080 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:50.540611 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:50.540615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:50.540669 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:50.540743 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:50.542146 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.542730 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.544404 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.544764 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.544853 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.548833 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:50.548839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:50.548844 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:50.548880 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:50.552486 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.553052 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.553056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.553057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.553148 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.553569 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.553572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.553574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.553798 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:50.553867 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:50.554488 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 10:44:50.554641 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:44:50.556141 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.538781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.10.96.193:49253)
0927 10:44:50.539301 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:50.539356 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:50.540365 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:50.540911 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:50.540920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:50.541350 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:50.541994 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.542011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.542830 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.542833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.544255 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.544269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.545051 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.545055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.545258 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.546014 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:50.546037 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:50.548610 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:50.552600 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.552606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.552620 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.552947 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.553253 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.553256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.553258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.553421 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.553943 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:50.553949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:50.554230 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 10:44:50.554940 (+ 710us) client_negotiation.cc:240] Negotiation successful
0927 10:44:50.555269 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":144,"threads_started":1}
I20260927 10:44:50.557396 11945 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43335
I20260927 10:44:50.557641 11945 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:50.558244 11945 heartbeater.cc:511] Master 127.10.96.254:43335 requested a full tablet report, sending...
I20260927 10:44:50.560412 11741 ts_manager.cc:194] Registered new tserver with Master: 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329)
I20260927 10:44:50.561986 11741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:49253
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:44:50.570025 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.559686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:51163 (local address 127.10.96.254:43335)
0927 10:44:50.561602 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:50.561608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:50.561624 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:50.561663 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:50.561668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:50.561729 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:50.561820 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:50.562494 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.563317 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.564391 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.564762 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.564846 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.566398 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:50.566403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:50.566405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:50.566433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:50.567885 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.568453 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.568459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.568462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.568549 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.568922 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.568926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.568929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.569179 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:50.569263 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:50.569708 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 10:44:50.569837 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1821}
I20260927 10:44:50.570716 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.559661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.10.96.193:51163)
0927 10:44:50.559936 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:50.559951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:50.560035 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:50.561954 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:50.561958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:50.562194 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:50.562346 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.562355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.563472 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.563476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.564258 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.564268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.565021 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.565025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.565145 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.565695 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:50.565713 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:50.566234 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:50.567994 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.567998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.568002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.568322 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.568692 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.568698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.568701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.568798 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.569341 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:50.569346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:50.569486 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 10:44:50.570249 (+ 763us) client_negotiation.cc:240] Negotiation successful
0927 10:44:50.570436 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:44:50.584365 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43335
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:50.693583 11954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:50.693812 11954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:50.693883 11954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:50.698001 11954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:50.698057 11954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:50.698119 11954 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:50.698189 11954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:44:50.703511 11954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:50.704532 11954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:50.705560 11954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:50.714633 11960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.714682 11962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.714681 11959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:50.714880 11954 server_base.cc:1034] running on GCE node
I20260927 10:44:50.715411 11954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:50.716037 11954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:50.717211 11954 hybrid_clock.cc:648] HybridClock initialized: now 1790505890717189 us; error 61 us; skew 500 ppm
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:50.719987 11954 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:44:50.720868 11954 webserver.cc:460] Webserver started at http://127.10.96.194:39485/ using document root <none> and password file <none>
I20260927 10:44:50.721407 11954 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:50.721516 11954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:50.721735 11954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:50.723675 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "19361e3753224eb08824c1ed1c3dc77a"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "3b7de3f18718472c952390888a621e8b"
server_key_iv: "6fcf55d5548faaf72572f2776c8ba7d1"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.724233 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "19361e3753224eb08824c1ed1c3dc77a"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "3b7de3f18718472c952390888a621e8b"
server_key_iv: "6fcf55d5548faaf72572f2776c8ba7d1"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.727787 11954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:44:50.730127 11969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.731331 11954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:50.731452 11954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "19361e3753224eb08824c1ed1c3dc77a"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "3b7de3f18718472c952390888a621e8b"
server_key_iv: "6fcf55d5548faaf72572f2776c8ba7d1"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.731552 11954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:50.764611 11954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:50.767823 11954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:50.768025 11954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:50.769429 11954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:50.769534 11954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.769605 11954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:50.769651 11954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:50.780498 11954 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:36987
I20260927 10:44:50.780702 12084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:36987 every 8 connection(s)
I20260927 10:44:50.781421 11954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:50.783383 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.770342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:48121 (local address 127.10.96.254:43335)
0927 10:44:50.770463 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:50.770467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:50.771530 (+ 1063us) server_negotiation.cc:407] Connection header received
0927 10:44:50.772583 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:50.772588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:50.772651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:50.772706 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:50.774154 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.775068 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.776295 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.776606 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.776721 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.779479 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:50.779483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:50.779485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:50.779517 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:50.781172 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.781759 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.781764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.781766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.781876 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.782268 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.782271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.782273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.782485 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:50.782588 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:50.783130 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 10:44:50.783262 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:44:50.784147 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.770915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.10.96.194:48121)
0927 10:44:50.771391 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:50.771432 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:50.772322 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:50.773017 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:50.773023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:50.773334 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:50.773938 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.773953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.775201 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.775207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.776175 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.776184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.776763 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.776767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.776938 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.777557 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:50.777576 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:50.779318 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:50.781282 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.781288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.781301 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.781659 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.782000 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.782003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.782005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.782161 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.782666 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:50.782676 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:50.782912 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 10:44:50.783071 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 10:44:50.783304 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260927 10:44:50.785404 12085 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43335
I20260927 10:44:50.785578 12085 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:50.786028 12085 heartbeater.cc:511] Master 127.10.96.254:43335 requested a full tablet report, sending...
I20260927 10:44:50.786980 11742 ts_manager.cc:194] Registered new tserver with Master: 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194:36987)
I20260927 10:44:50.787704 11742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:48121
I20260927 10:44:50.790431 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 11954
I20260927 10:44:50.790545 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:50.790855 10627 external_mini_cluster.cc:1428] Setting key 1157c9dbad326d06bf09baa2a04834a1
I20260927 10:44:50.794554 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.787404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.10.96.194:55761)
0927 10:44:50.787660 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:50.787674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:50.787781 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:50.788045 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:50.788048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:50.788244 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:50.788374 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.788380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.789104 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.789107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.789790 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.789798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.790257 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.790261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.790353 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.790865 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:50.790877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:50.791262 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:50.792967 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.792973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.792978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.793235 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.793501 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.793507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.793512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.793611 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.794148 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:50.794150 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:50.794231 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 10:44:50.794314 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:44:50.794413 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260927 10:44:50.794682 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.787453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:55761 (local address 127.10.96.254:43335)
0927 10:44:50.787578 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:50.787583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:50.787750 (+ 167us) server_negotiation.cc:407] Connection header received
0927 10:44:50.787902 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:50.787904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:50.787943 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:50.788030 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:50.788481 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.788989 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.789902 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.790125 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.790237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.791400 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:50.791403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:50.791405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:50.791425 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:50.792861 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.793326 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.793328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.793330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.793395 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.793739 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.793744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.793748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.794043 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:50.794143 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:50.794389 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 10:44:50.794511 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:44:50.816129 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43335
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:50.925530 12094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:50.925827 12094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:50.925902 12094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:50.929870 12094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:50.929955 12094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:50.930027 12094 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:50.930104 12094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:44:50.935343 12094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:50.936388 12094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:50.937404 12094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:50.946772 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.946775 12100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:50.946791 12099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:50.946842 12094 server_base.cc:1034] running on GCE node
I20260927 10:44:50.947371 12094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:50.948050 12094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:50.949235 12094 hybrid_clock.cc:648] HybridClock initialized: now 1790505890949218 us; error 33 us; skew 500 ppm
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:50.951954 12094 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:44:50.952719 12094 webserver.cc:460] Webserver started at http://127.10.96.195:32899/ using document root <none> and password file <none>
I20260927 10:44:50.953223 12094 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:50.953279 12094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:50.953436 12094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:50.955473 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "adbbdca7f04e4186a1372f42425ef0aa"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "8db9ee2b088e445c0709ec324a98e622"
server_key_iv: "c72e75dbd7357e546da33227882a5b98"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.955979 12094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "adbbdca7f04e4186a1372f42425ef0aa"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "8db9ee2b088e445c0709ec324a98e622"
server_key_iv: "c72e75dbd7357e546da33227882a5b98"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.959836 12094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 10:44:50.962211 12109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.963409 12094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:50.963521 12094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "adbbdca7f04e4186a1372f42425ef0aa"
format_stamp: "Formatted at 2026-09-27 10:44:50 on dist-test-slave-wmg0"
server_key: "8db9ee2b088e445c0709ec324a98e622"
server_key_iv: "c72e75dbd7357e546da33227882a5b98"
server_key_version: "encryptionkey@0"
I20260927 10:44:50.963624 12094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:50.978626 12094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:50.982723 12094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:50.982899 12094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:50.984300 12094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:50.984349 12094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:50.984390 12094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:50.984438 12094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:50 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:50.995011 12094 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:36067
I20260927 10:44:50.995784 12224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:36067 every 8 connection(s)
I20260927 10:44:50.996138 12094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:50.997786 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.985012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:37491 (local address 127.10.96.254:43335)
0927 10:44:50.985149 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:50.985154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:50.986140 (+ 986us) server_negotiation.cc:407] Connection header received
0927 10:44:50.987188 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:50.987192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:50.987250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:50.987333 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:50.988762 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.989639 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.990786 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.991068 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.991175 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.993871 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:50.993878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:50.993885 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:50.993917 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:50.995571 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.996085 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.996092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.996097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.996193 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:50.996542 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:50.996547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:50.996549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:50.996793 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:50.996895 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:50.997415 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 10:44:50.997588 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:44:50.998610 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:50.985516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.10.96.195:37491)
0927 10:44:50.985983 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:50.986031 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:50.986981 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:50.987497 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:50.987506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:50.987996 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:50.988613 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.988634 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.989753 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.989758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.990648 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:50.990658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:50.991237 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:50.991241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:50.991454 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:50.992081 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:50.992100 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:50.993702 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:50.995684 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.995689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.995699 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.995992 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.996289 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:50.996292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:50.996294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:50.996431 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:50.996913 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:50.996919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:50.997214 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 10:44:50.997371 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:44:50.997704 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20260927 10:44:50.999920 12225 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43335
I20260927 10:44:51.000064 12225 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:51.000594 12225 heartbeater.cc:511] Master 127.10.96.254:43335 requested a full tablet report, sending...
I20260927 10:44:51.001677 11742 ts_manager.cc:194] Registered new tserver with Master: adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.001843 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 12094
I20260927 10:44:51.001962 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:51.002269 10627 external_mini_cluster.cc:1428] Setting key a793c40122a46e762d23c61860b2cc08
I20260927 10:44:51.002440 11742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:37491
I20260927 10:44:51.004395 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:44:51.009698 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.002713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:58003 (local address 127.10.96.254:43335)
0927 10:44:51.002827 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.002830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.002860 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:44:51.002894 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.002897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.002941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.003015 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:51.003548 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.004257 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.005092 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.005321 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.005424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.006665 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:51.006668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:51.006670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:51.006689 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:51.008057 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.008529 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.008532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.008533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.008600 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.008889 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.008891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.008892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.009033 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:51.009104 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.009381 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.009512 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:44:51.009755 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.002005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.10.96.195:58003)
0927 10:44:51.002285 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.002299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.002378 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.003112 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.003115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.003333 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:51.003444 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.003450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.004380 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.004383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.004981 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.004989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.005423 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.005426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.005539 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.006050 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:51.006068 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:51.006520 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:51.008158 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:51.008165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:51.008169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:51.008436 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:51.008684 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:51.008696 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:51.008701 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:51.008803 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:51.009120 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:51.009125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:51.009230 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.009343 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.009508 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":138,"threads_started":1}
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[11697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:51.030189 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.022412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.10.96.254:43335)
0927 10:44:51.022535 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.022538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.022550 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:44:51.022587 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.022589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.022625 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.022708 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:51.023377 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.023890 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.024685 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.024907 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.024997 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.027294 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:51.027297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:51.027299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:51.027320 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:51.028552 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.028922 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.028924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.028925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.028994 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.029256 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.029259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.029260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.029446 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:51.029639 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.029889 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.029981 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:44:51.032913 11742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:51.036234 11742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:51.050045 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.044391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39329 (local address 127.0.0.1:36882)
0927 10:44:51.044690 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.044742 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.044866 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.045648 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.045656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.045702 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.045857 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.045865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.047190 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.047193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.048408 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.048417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.049447 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.049451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.049592 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.049636 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.049747 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.049867 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":3,"spinlock_wait_cycles":5760,"thread_start_us":80,"threads_started":1}
I20260927 10:44:51.050236 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.045119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:36067 (local address 127.0.0.1:46736)
0927 10:44:51.045694 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.045707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.045786 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.046434 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.046437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.046448 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.046517 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.046522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.047789 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.047792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.048925 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.048936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.049776 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.049780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.049880 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.049909 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.049987 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.050058 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":2,"thread_start_us":125,"threads_started":1}
I20260927 10:44:51.050256 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.045265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.10.96.195:36067)
0927 10:44:51.046016 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.046031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.046053 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:44:51.046123 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.046129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.046283 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.046649 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.046748 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.047682 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.049017 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.049653 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.049772 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.049919 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.050014 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.050119 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":96,"threads_started":1}
I20260927 10:44:51.050658 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.044540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.10.96.193:39329)
0927 10:44:51.044940 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.044955 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.044981 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:44:51.045072 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.045079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.045293 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.045450 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.045985 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.047077 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.048536 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.049326 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.049433 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.049642 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.050406 (+ 764us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.050500 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":83,"threads_started":1}
I20260927 10:44:51.050776 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.044699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:36987 (local address 127.0.0.1:35558)
0927 10:44:51.045600 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.045614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.045707 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.046378 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.046381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.046401 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.046508 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.046515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.047614 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.047620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.048847 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.048857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.050265 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.050269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.050480 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.050502 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.050573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.050640 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":6,"spinlock_wait_cycles":17408,"thread_start_us":91,"threads_started":1}
I20260927 10:44:51.051152 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.045160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.10.96.194:36987)
0927 10:44:51.045950 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.045962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.045982 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:44:51.046053 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.046058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.046239 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.046381 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.046619 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.047511 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.048996 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.049826 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.049923 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.050086 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.050944 (+ 858us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.051035 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":108,"threads_started":1}
I20260927 10:44:51.053319 11879 tablet_service.cc:1467] Processing CreateTablet for tablet e2b1eba114594d8e9680c1c9a29878ff (DEFAULT_TABLE table=test-table [id=515586cff5c14f48bebcf136fd3916cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:51.053252 12159 tablet_service.cc:1467] Processing CreateTablet for tablet e2b1eba114594d8e9680c1c9a29878ff (DEFAULT_TABLE table=test-table [id=515586cff5c14f48bebcf136fd3916cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:51.053743 12019 tablet_service.cc:1467] Processing CreateTablet for tablet e2b1eba114594d8e9680c1c9a29878ff (DEFAULT_TABLE table=test-table [id=515586cff5c14f48bebcf136fd3916cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:51.054303 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b1eba114594d8e9680c1c9a29878ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.054320 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b1eba114594d8e9680c1c9a29878ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.054572 12019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b1eba114594d8e9680c1c9a29878ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.061823 12248 tablet_bootstrap.cc:492] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Bootstrap starting.
I20260927 10:44:51.061823 12249 tablet_bootstrap.cc:492] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Bootstrap starting.
I20260927 10:44:51.062565 12250 tablet_bootstrap.cc:492] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Bootstrap starting.
I20260927 10:44:51.064172 12249 tablet_bootstrap.cc:654] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.064173 12248 tablet_bootstrap.cc:654] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.064318 12250 tablet_bootstrap.cc:654] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.064967 12250 log.cc:824] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:51.065018 12249 log.cc:824] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:51.065018 12248 log.cc:824] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:51.066941 12248 tablet_bootstrap.cc:492] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: No bootstrap required, opened a new log
I20260927 10:44:51.066941 12249 tablet_bootstrap.cc:492] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: No bootstrap required, opened a new log
I20260927 10:44:51.066995 12250 tablet_bootstrap.cc:492] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: No bootstrap required, opened a new log
I20260927 10:44:51.067132 12249 ts_tablet_manager.cc:1397] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:44:51.067128 12248 ts_tablet_manager.cc:1397] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:44:51.067219 12250 ts_tablet_manager.cc:1397] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:44:51.070134 12250 raft_consensus.cc:348] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.070363 12250 raft_consensus.cc:374] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.070418 12250 raft_consensus.cc:729] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19361e3753224eb08824c1ed1c3dc77a, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.070835 12250 consensus_queue.cc:260] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.070945 12248 raft_consensus.cc:348] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.070924 12249 raft_consensus.cc:348] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.071174 12248 raft_consensus.cc:374] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.071174 12249 raft_consensus.cc:374] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.071231 12248 raft_consensus.cc:729] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f27f86c137640608976876a64f6d39e, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.071231 12249 raft_consensus.cc:729] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbbdca7f04e4186a1372f42425ef0aa, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.071632 12249 consensus_queue.cc:260] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.071633 12248 consensus_queue.cc:260] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.071785 12250 ts_tablet_manager.cc:1428] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:44:51.072125 12085 heartbeater.cc:503] Master 127.10.96.254:43335 was elected leader, sending a full tablet report...
I20260927 10:44:51.072170 11945 heartbeater.cc:503] Master 127.10.96.254:43335 was elected leader, sending a full tablet report...
I20260927 10:44:51.072515 12248 ts_tablet_manager.cc:1428] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:44:51.073340 12249 ts_tablet_manager.cc:1428] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260927 10:44:51.073760 12225 heartbeater.cc:503] Master 127.10.96.254:43335 was elected leader, sending a full tablet report...
I20260927 10:44:51.176297 12254 raft_consensus.cc:482] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:51.176501 12254 raft_consensus.cc:504] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.177671 12254 leader_election.cc:288] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329), adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.181236 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.178088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:52043 (local address 127.10.96.193:39329)
0927 10:44:51.178242 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.178247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.178263 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:44:51.178342 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.178346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.178405 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.178476 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.178790 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.179309 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.180333 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.180713 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.180834 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.180888 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.181048 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.181112 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:44:51.181236 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.177953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39329 (local address 127.10.96.194:52043)
0927 10:44:51.178094 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.178112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.178241 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.178542 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.178545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.178564 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.178667 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.178671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.179416 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.179421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.180190 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.180198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.180821 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.180825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.180928 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.180953 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.181034 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.181104 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:44:51.181929 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b1eba114594d8e9680c1c9a29878ff" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f27f86c137640608976876a64f6d39e" is_pre_election: true
I20260927 10:44:51.182080 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.178392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:37637 (local address 127.10.96.195:36067)
0927 10:44:51.178525 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.178530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.178678 (+ 148us) server_negotiation.cc:407] Connection header received
0927 10:44:51.178797 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.178800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.178844 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.178922 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.179178 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.179660 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.181054 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.181584 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.181707 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.181748 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.181923 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.181972 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:44:51.182102 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.178250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:36067 (local address 127.10.96.194:37637)
0927 10:44:51.178595 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.178606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.178711 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.178932 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.178935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.178949 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.179070 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.179075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.179778 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.179781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.180934 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.180942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.181720 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.181724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.181812 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.181832 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.181910 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.181978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260927 10:44:51.182272 11899 raft_consensus.cc:2454] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 0.
I20260927 10:44:51.182621 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b1eba114594d8e9680c1c9a29878ff" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbbdca7f04e4186a1372f42425ef0aa" is_pre_election: true
I20260927 10:44:51.182782 11970 leader_election.cc:302] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19361e3753224eb08824c1ed1c3dc77a, 4f27f86c137640608976876a64f6d39e; no voters:
I20260927 10:44:51.182911 12179 raft_consensus.cc:2454] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 0.
I20260927 10:44:51.183051 12254 raft_consensus.cc:2781] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:51.183162 12254 raft_consensus.cc:482] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:51.183224 12254 raft_consensus.cc:3037] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.184223 12254 raft_consensus.cc:504] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.184500 12254 leader_election.cc:288] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 election: Requested vote from peers 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329), adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.184821 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b1eba114594d8e9680c1c9a29878ff" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f27f86c137640608976876a64f6d39e"
I20260927 10:44:51.184870 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2b1eba114594d8e9680c1c9a29878ff" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbbdca7f04e4186a1372f42425ef0aa"
I20260927 10:44:51.184968 11899 raft_consensus.cc:3037] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.185001 12179 raft_consensus.cc:3037] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.186028 11899 raft_consensus.cc:2454] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 1.
I20260927 10:44:51.186144 12179 raft_consensus.cc:2454] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 1.
I20260927 10:44:51.186311 11970 leader_election.cc:302] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19361e3753224eb08824c1ed1c3dc77a, 4f27f86c137640608976876a64f6d39e; no voters:
I20260927 10:44:51.186482 12254 raft_consensus.cc:2781] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:51.186712 12254 raft_consensus.cc:686] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 1 LEADER]: Becoming Leader. State: Replica: 19361e3753224eb08824c1ed1c3dc77a, State: Running, Role: LEADER
I20260927 10:44:51.187012 12254 consensus_queue.cc:237] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } }
I20260927 10:44:51.189895 11740 catalog_manager.cc:5495] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a reported cstate change: term changed from 0 to 1, leader changed from <none> to 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194). New cstate: current_term: 1 leader_uuid: "19361e3753224eb08824c1ed1c3dc77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } health_report { overall_health: HEALTHY } } }
I20260927 10:44:51.211740 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.208356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.10.96.194:36987)
0927 10:44:51.208476 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.208480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.208497 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:44:51.208643 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.208645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.208706 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.208799 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:51.209006 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.209550 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.210386 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.210607 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.210689 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.211007 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.211416 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.211506 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.211575 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.211627 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:44:51.217110 11899 raft_consensus.cc:1261] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Refusing update from remote peer 19361e3753224eb08824c1ed1c3dc77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:51.217382 12179 raft_consensus.cc:1261] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Refusing update from remote peer 19361e3753224eb08824c1ed1c3dc77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:51.217859 12254 consensus_queue.cc:1035] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:51.218050 12258 consensus_queue.cc:1035] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:51.225652 12263 mvcc.cc:205] Tried to move back new op lower bound from 7333912130418081792 to 7333912130304262144. Current Snapshot: MvccSnapshot[applied={T|T < 7333912130418081792}]
I20260927 10:44:51.228523 12261 mvcc.cc:205] Tried to move back new op lower bound from 7333912130418081792 to 7333912130304262144. Current Snapshot: MvccSnapshot[applied={T|T < 7333912130418081792 or (T in {7333912130418081792})}]
I20260927 10:44:51.235601 11740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:51.235764 11740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:51.237985 11740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: Sending DeleteTablet for 3 replicas of tablet e2b1eba114594d8e9680c1c9a29878ff
I20260927 10:44:51.238696 12019 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b1eba114594d8e9680c1c9a29878ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:51 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:35558
I20260927 10:44:51.238729 12159 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b1eba114594d8e9680c1c9a29878ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:51 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:46736
I20260927 10:44:51.238786 11879 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b1eba114594d8e9680c1c9a29878ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:51 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:36882
I20260927 10:44:51.239255 12273 tablet_replica.cc:330] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: stopping tablet replica
I20260927 10:44:51.239493 12272 tablet_replica.cc:330] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: stopping tablet replica
I20260927 10:44:51.239557 12273 raft_consensus.cc:2229] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:51.239861 12273 raft_consensus.cc:2258] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:51.239969 12271 tablet_replica.cc:330] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: stopping tablet replica
I20260927 10:44:51.240060 11740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:44:51.240309 12272 raft_consensus.cc:2229] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 10:44:51.240477 11740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:51.240578 12272 raft_consensus.cc:2258] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:51.240972 12271 raft_consensus.cc:2229] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:51.241307 12271 raft_consensus.cc:2258] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:51.242153 12272 ts_tablet_manager.cc:1905] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:51.242440 12273 ts_tablet_manager.cc:1905] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:51.245030 12272 ts_tablet_manager.cc:1918] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:51.245033 12273 ts_tablet_manager.cc:1918] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:51.245131 12273 log.cc:1196] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/e2b1eba114594d8e9680c1c9a29878ff
I20260927 10:44:51.245257 12272 log.cc:1196] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/e2b1eba114594d8e9680c1c9a29878ff
I20260927 10:44:51.245518 12273 ts_tablet_manager.cc:1939] T e2b1eba114594d8e9680c1c9a29878ff P 4f27f86c137640608976876a64f6d39e: Deleting consensus metadata
I20260927 10:44:51.245740 12272 ts_tablet_manager.cc:1939] T e2b1eba114594d8e9680c1c9a29878ff P adbbdca7f04e4186a1372f42425ef0aa: Deleting consensus metadata
I20260927 10:44:51.246340 12019 tablet_service.cc:1467] Processing CreateTablet for tablet e98fc41b6a684ee8b3fb7082bdcdbe45 (DEFAULT_TABLE table=test-table [id=dd1c76f8a41042bdafd52b1959af33d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:51.246609 12019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98fc41b6a684ee8b3fb7082bdcdbe45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.247731 12271 ts_tablet_manager.cc:1905] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:51.248632 12250 tablet_bootstrap.cc:492] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Bootstrap starting.
I20260927 10:44:51.248896 11726 catalog_manager.cc:4841] TS adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067): tablet e2b1eba114594d8e9680c1c9a29878ff (table test-table [id=515586cff5c14f48bebcf136fd3916cf]) successfully deleted
I20260927 10:44:51.249051 11726 catalog_manager.cc:4841] TS 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329): tablet e2b1eba114594d8e9680c1c9a29878ff (table test-table [id=515586cff5c14f48bebcf136fd3916cf]) successfully deleted
I20260927 10:44:51.249642 12250 tablet_bootstrap.cc:654] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.249750 11879 tablet_service.cc:1467] Processing CreateTablet for tablet e98fc41b6a684ee8b3fb7082bdcdbe45 (DEFAULT_TABLE table=test-table [id=dd1c76f8a41042bdafd52b1959af33d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:51.249994 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98fc41b6a684ee8b3fb7082bdcdbe45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.250115 12271 ts_tablet_manager.cc:1918] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:44:51.250190 12271 log.cc:1196] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/e2b1eba114594d8e9680c1c9a29878ff
I20260927 10:44:51.250506 12271 ts_tablet_manager.cc:1939] T e2b1eba114594d8e9680c1c9a29878ff P 19361e3753224eb08824c1ed1c3dc77a: Deleting consensus metadata
I20260927 10:44:51.251530 11727 catalog_manager.cc:4841] TS 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194:36987): tablet e2b1eba114594d8e9680c1c9a29878ff (table test-table [id=515586cff5c14f48bebcf136fd3916cf]) successfully deleted
I20260927 10:44:51.251904 12248 tablet_bootstrap.cc:492] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Bootstrap starting.
I20260927 10:44:51.252228 12159 tablet_service.cc:1467] Processing CreateTablet for tablet e98fc41b6a684ee8b3fb7082bdcdbe45 (DEFAULT_TABLE table=test-table [id=dd1c76f8a41042bdafd52b1959af33d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:51.252411 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e98fc41b6a684ee8b3fb7082bdcdbe45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.252781 12248 tablet_bootstrap.cc:654] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.254581 12249 tablet_bootstrap.cc:492] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Bootstrap starting.
I20260927 10:44:51.255558 12249 tablet_bootstrap.cc:654] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.255982 12250 tablet_bootstrap.cc:492] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: No bootstrap required, opened a new log
I20260927 10:44:51.256067 12250 ts_tablet_manager.cc:1397] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 10:44:51.256421 12248 tablet_bootstrap.cc:492] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: No bootstrap required, opened a new log
I20260927 10:44:51.256491 12248 ts_tablet_manager.cc:1397] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 10:44:51.256511 12250 raft_consensus.cc:348] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.256610 12250 raft_consensus.cc:374] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.256635 12250 raft_consensus.cc:729] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19361e3753224eb08824c1ed1c3dc77a, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.256760 12250 consensus_queue.cc:260] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.256887 12248 raft_consensus.cc:348] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.256983 12248 raft_consensus.cc:374] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.257015 12248 raft_consensus.cc:729] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f27f86c137640608976876a64f6d39e, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.257138 12248 consensus_queue.cc:260] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.257709 12249 tablet_bootstrap.cc:492] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: No bootstrap required, opened a new log
I20260927 10:44:51.257779 12249 ts_tablet_manager.cc:1397] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:51.258183 12249 raft_consensus.cc:348] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.258289 12249 raft_consensus.cc:374] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.258322 12249 raft_consensus.cc:729] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbbdca7f04e4186a1372f42425ef0aa, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.258431 12249 consensus_queue.cc:260] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.258652 12249 ts_tablet_manager.cc:1428] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:44:51.258846 12250 ts_tablet_manager.cc:1428] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:51.259169 12248 ts_tablet_manager.cc:1428] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20260927 10:44:51.282538 12086 tablet.cc:2369] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:44:51.302241 11946 tablet.cc:2369] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:51.322068 12268 raft_consensus.cc:482] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:51.322211 12268 raft_consensus.cc:504] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.322620 12268 leader_election.cc:288] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329), adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.322976 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fc41b6a684ee8b3fb7082bdcdbe45" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f27f86c137640608976876a64f6d39e" is_pre_election: true
I20260927 10:44:51.323129 11899 raft_consensus.cc:2454] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 0.
I20260927 10:44:51.323104 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fc41b6a684ee8b3fb7082bdcdbe45" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbbdca7f04e4186a1372f42425ef0aa" is_pre_election: true
I20260927 10:44:51.323262 12179 raft_consensus.cc:2454] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 0.
I20260927 10:44:51.323410 11970 leader_election.cc:302] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19361e3753224eb08824c1ed1c3dc77a, 4f27f86c137640608976876a64f6d39e; no voters:
I20260927 10:44:51.323561 12268 raft_consensus.cc:2781] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:51.323607 12268 raft_consensus.cc:482] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:51.323644 12268 raft_consensus.cc:3037] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.324573 12268 raft_consensus.cc:504] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.324909 12268 leader_election.cc:288] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 election: Requested vote from peers 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329), adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.325165 11899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fc41b6a684ee8b3fb7082bdcdbe45" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f27f86c137640608976876a64f6d39e"
I20260927 10:44:51.325274 11899 raft_consensus.cc:3037] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.325304 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e98fc41b6a684ee8b3fb7082bdcdbe45" candidate_uuid: "19361e3753224eb08824c1ed1c3dc77a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbbdca7f04e4186a1372f42425ef0aa"
I20260927 10:44:51.325428 12179 raft_consensus.cc:3037] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.326321 12179 raft_consensus.cc:2454] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 1.
I20260927 10:44:51.326402 11899 raft_consensus.cc:2454] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19361e3753224eb08824c1ed1c3dc77a in term 1.
I20260927 10:44:51.326673 11970 leader_election.cc:302] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19361e3753224eb08824c1ed1c3dc77a, 4f27f86c137640608976876a64f6d39e; no voters:
I20260927 10:44:51.326930 12268 raft_consensus.cc:2781] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:51.327016 12268 raft_consensus.cc:686] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 1 LEADER]: Becoming Leader. State: Replica: 19361e3753224eb08824c1ed1c3dc77a, State: Running, Role: LEADER
I20260927 10:44:51.327160 12268 consensus_queue.cc:237] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.328756 11741 catalog_manager.cc:5495] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a reported cstate change: term changed from 0 to 1, leader changed from <none> to 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194). New cstate: current_term: 1 leader_uuid: "19361e3753224eb08824c1ed1c3dc77a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505890, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:51.369637 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.359946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43335 (local address 127.0.0.1:59080)
0927 10:44:51.360313 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.360325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.360428 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.360719 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.360721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.361096 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:51.361193 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.361200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.362282 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.362285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.362974 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.362985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.363755 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.363758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.363845 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.364516 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:51.364527 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:51.366133 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:51.368070 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:51.368078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:51.368082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:51.368363 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:51.368652 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:51.368654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:51.368655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:51.368726 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:51.369141 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:51.369147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:51.369244 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.369334 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.369438 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":113,"threads_started":1}
I20260927 10:44:51.369779 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.360002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.10.96.254:43335)
0927 10:44:51.360138 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.360145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.360413 (+ 268us) server_negotiation.cc:407] Connection header received
0927 10:44:51.360518 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.360522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.360574 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.360681 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:51.361319 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.362150 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.363107 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.363474 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.363566 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.366274 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:51.366277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:51.366280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:51.366305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:51.367967 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.368464 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.368468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.368470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.368557 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.368813 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.368816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.368818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.369038 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:51.369143 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.369419 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.369575 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:51.374281 11741 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:59080:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:44:51.375013 11741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:51.379822 11879 tablet_service.cc:1467] Processing CreateTablet for tablet 75d15b53912c4ecabdbf35dee71a015b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4725b7e90be47d3a06e19d4f27d51ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:51.380056 12019 tablet_service.cc:1467] Processing CreateTablet for tablet 75d15b53912c4ecabdbf35dee71a015b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4725b7e90be47d3a06e19d4f27d51ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:51.380218 11879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d15b53912c4ecabdbf35dee71a015b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.380411 12019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d15b53912c4ecabdbf35dee71a015b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.381839 12159 tablet_service.cc:1467] Processing CreateTablet for tablet 75d15b53912c4ecabdbf35dee71a015b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c4725b7e90be47d3a06e19d4f27d51ef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:51.382177 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d15b53912c4ecabdbf35dee71a015b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:51.382957 12250 tablet_bootstrap.cc:492] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: Bootstrap starting.
I20260927 10:44:51.383198 12248 tablet_bootstrap.cc:492] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: Bootstrap starting.
I20260927 10:44:51.383931 12250 tablet_bootstrap.cc:654] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.384128 12248 tablet_bootstrap.cc:654] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.385519 12249 tablet_bootstrap.cc:492] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: Bootstrap starting.
I20260927 10:44:51.385716 12250 tablet_bootstrap.cc:492] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: No bootstrap required, opened a new log
I20260927 10:44:51.385787 12250 ts_tablet_manager.cc:1397] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:51.386247 12250 raft_consensus.cc:348] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.386351 12250 raft_consensus.cc:374] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.386389 12250 raft_consensus.cc:729] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19361e3753224eb08824c1ed1c3dc77a, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.386524 12249 tablet_bootstrap.cc:654] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:51.386533 12250 consensus_queue.cc:260] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.386827 12250 ts_tablet_manager.cc:1428] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:51.387081 12268 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.387215 12268 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.387940 12248 tablet_bootstrap.cc:492] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: No bootstrap required, opened a new log
I20260927 10:44:51.388036 12248 ts_tablet_manager.cc:1397] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 10:44:51.388536 12248 raft_consensus.cc:348] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.388612 12249 tablet_bootstrap.cc:492] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: No bootstrap required, opened a new log
I20260927 10:44:51.388643 12248 raft_consensus.cc:374] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.388674 12248 raft_consensus.cc:729] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f27f86c137640608976876a64f6d39e, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.388696 12249 ts_tablet_manager.cc:1397] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:51.388797 12248 consensus_queue.cc:260] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.389048 12248 ts_tablet_manager.cc:1428] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:51.389110 12249 raft_consensus.cc:348] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.389205 12249 raft_consensus.cc:374] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:51.389237 12249 raft_consensus.cc:729] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbbdca7f04e4186a1372f42425ef0aa, State: Initialized, Role: FOLLOWER
I20260927 10:44:51.389276 12255 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.389389 12255 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.389348 12249 consensus_queue.cc:260] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.389626 12249 ts_tablet_manager.cc:1428] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:51.389658 12256 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.389796 12256 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.474187 12255 raft_consensus.cc:482] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:51.474327 12255 raft_consensus.cc:504] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.475447 12255 leader_election.cc:288] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194:36987), adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.479425 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.475756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:45793 (local address 127.10.96.195:36067)
0927 10:44:51.475890 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.475895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.476109 (+ 214us) server_negotiation.cc:407] Connection header received
0927 10:44:51.476267 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.476270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.476316 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.476395 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.476688 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.477182 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.478470 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.478859 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.478979 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.479037 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.479229 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.479298 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:44:51.479458 12288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.475693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:36067 (local address 127.10.96.193:45793)
0927 10:44:51.476017 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.476028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.476156 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.476407 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.476410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.476425 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.476585 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.476590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.477289 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.477294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.478342 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.478351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.478970 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.478973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.479087 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.479123 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.479215 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.479292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":79,"spinlock_wait_cycles":4480,"thread_start_us":112,"threads_started":1}
I20260927 10:44:51.479452 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.475793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55271 (local address 127.10.96.194:36987)
0927 10:44:51.475946 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.475950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.476089 (+ 139us) server_negotiation.cc:407] Connection header received
0927 10:44:51.476243 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.476246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.476286 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.476366 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:51.476690 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.477180 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.478476 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.478877 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.479000 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.479050 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.479251 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.479326 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:44:51.479475 12287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.475647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:36987 (local address 127.10.96.193:55271)
0927 10:44:51.476006 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.476017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.476141 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.476378 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.476380 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.476395 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:51.476591 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.476596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.477281 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.477287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.478337 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.478360 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.479000 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.479004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.479138 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.479156 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.479236 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.479326 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":182,"spinlock_wait_cycles":6656,"thread_start_us":119,"threads_started":1}
I20260927 10:44:51.479918 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d15b53912c4ecabdbf35dee71a015b" candidate_uuid: "4f27f86c137640608976876a64f6d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbbdca7f04e4186a1372f42425ef0aa" is_pre_election: true
I20260927 10:44:51.480065 12179 raft_consensus.cc:2454] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f27f86c137640608976876a64f6d39e in term 0.
I20260927 10:44:51.480257 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d15b53912c4ecabdbf35dee71a015b" candidate_uuid: "4f27f86c137640608976876a64f6d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19361e3753224eb08824c1ed1c3dc77a" is_pre_election: true
I20260927 10:44:51.480415 11830 leader_election.cc:302] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f27f86c137640608976876a64f6d39e, adbbdca7f04e4186a1372f42425ef0aa; no voters:
I20260927 10:44:51.480468 12039 raft_consensus.cc:2454] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f27f86c137640608976876a64f6d39e in term 0.
I20260927 10:44:51.480670 12255 raft_consensus.cc:2781] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:51.480738 12255 raft_consensus.cc:482] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:51.480780 12255 raft_consensus.cc:3037] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.481895 12255 raft_consensus.cc:504] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.482224 12255 leader_election.cc:288] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [CANDIDATE]: Term 1 election: Requested vote from peers 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194:36987), adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067)
I20260927 10:44:51.482549 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d15b53912c4ecabdbf35dee71a015b" candidate_uuid: "4f27f86c137640608976876a64f6d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19361e3753224eb08824c1ed1c3dc77a"
I20260927 10:44:51.482563 12179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d15b53912c4ecabdbf35dee71a015b" candidate_uuid: "4f27f86c137640608976876a64f6d39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbbdca7f04e4186a1372f42425ef0aa"
I20260927 10:44:51.482685 12179 raft_consensus.cc:3037] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.482698 12039 raft_consensus.cc:3037] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:51.483781 12179 raft_consensus.cc:2454] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f27f86c137640608976876a64f6d39e in term 1.
I20260927 10:44:51.483809 12039 raft_consensus.cc:2454] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f27f86c137640608976876a64f6d39e in term 1.
I20260927 10:44:51.484071 11831 leader_election.cc:302] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19361e3753224eb08824c1ed1c3dc77a, 4f27f86c137640608976876a64f6d39e; no voters:
I20260927 10:44:51.484228 12255 raft_consensus.cc:2781] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:51.484438 12255 raft_consensus.cc:686] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [term 1 LEADER]: Becoming Leader. State: Replica: 4f27f86c137640608976876a64f6d39e, State: Running, Role: LEADER
I20260927 10:44:51.484686 12255 consensus_queue.cc:237] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } }
I20260927 10:44:51.485893 12289 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: TxnStatusTablet state changed. Reason: New leader 4f27f86c137640608976876a64f6d39e. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.486014 12289 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:51.486501 12291 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:44:51.486656 12291 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:44:51.487097 11741 catalog_manager.cc:5495] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f27f86c137640608976876a64f6d39e (127.10.96.193). New cstate: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } health_report { overall_health: UNKNOWN } } }
W20260927 10:44:51.497169 12226 tablet.cc:2369] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:51.516278 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.512630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39329 (local address 127.0.0.1:36892)
0927 10:44:51.512804 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.512819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.512933 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.513267 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.513270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.513277 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:51.513401 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.513406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.514126 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.514130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.514738 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.514745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.515183 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.515187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.515277 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.515326 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.515993 (+ 667us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.516008 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.516078 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.516156 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 10:44:51.516299 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.512735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.10.96.193:39329)
0927 10:44:51.512869 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.512875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.512888 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:44:51.513092 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.513095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.513150 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.513244 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:51.513524 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.514021 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.514861 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.515072 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.515183 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.515474 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.515891 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.516020 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.516087 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.516136 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:44:51.519265 12179 raft_consensus.cc:1261] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Refusing update from remote peer 4f27f86c137640608976876a64f6d39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:51.519502 12039 raft_consensus.cc:1261] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Refusing update from remote peer 4f27f86c137640608976876a64f6d39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:51.519722 12289 consensus_queue.cc:1035] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:51.519996 12255 consensus_queue.cc:1035] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:51.521612 12256 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: TxnStatusTablet state changed. Reason: New leader 4f27f86c137640608976876a64f6d39e. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.521759 12256 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.523469 12289 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.523617 12289 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:51.523993 12289 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.524111 12289 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 4f27f86c137640608976876a64f6d39e: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:51.524616 12256 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.524725 12256 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P adbbdca7f04e4186a1372f42425ef0aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.524804 12268 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: TxnStatusTablet state changed. Reason: New leader 4f27f86c137640608976876a64f6d39e. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.524928 12268 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.524992 12254 tablet_replica.cc:439] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f27f86c137640608976876a64f6d39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 } } peers { permanent_uuid: "19361e3753224eb08824c1ed1c3dc77a" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36987 } } peers { permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 } } }
I20260927 10:44:51.525096 12254 tablet_replica.cc:442] T 75d15b53912c4ecabdbf35dee71a015b P 19361e3753224eb08824c1ed1c3dc77a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:51.538259 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.534784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:48333 (local address 127.10.96.193:39329)
0927 10:44:51.534935 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.534938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.535038 (+ 100us) server_negotiation.cc:407] Connection header received
0927 10:44:51.535176 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.535178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.535217 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.535279 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:51.535544 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.536050 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.537281 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.537526 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.537606 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.537891 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.537964 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.538016 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.538125 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.538163 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 10:44:51.538327 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.534644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39329 (local address 127.10.96.194:48333)
0927 10:44:51.534951 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.534966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.535091 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.535307 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.535310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.535319 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:51.535435 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.535442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.536173 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.536176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.537175 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.537185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.537623 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.537628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.537723 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.537773 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.538035 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.538041 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.538100 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.538180 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260927 10:44:51.544301 12179 raft_consensus.cc:1261] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Refusing update from remote peer 19361e3753224eb08824c1ed1c3dc77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:51.544396 11899 raft_consensus.cc:1261] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Refusing update from remote peer 19361e3753224eb08824c1ed1c3dc77a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:51.544692 12254 consensus_queue.cc:1035] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbbdca7f04e4186a1372f42425ef0aa" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:51.544821 12268 consensus_queue.cc:1035] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f27f86c137640608976876a64f6d39e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:51.559648 12262 mvcc.cc:205] Tried to move back new op lower bound from 7333912131762413568 to 7333912130877304832. Current Snapshot: MvccSnapshot[applied={T|T < 7333912131762413568}]
I20260927 10:44:51.578584 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.574422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:43751 (local address 127.10.96.194:36987)
0927 10:44:51.574563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.574568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.574747 (+ 179us) server_negotiation.cc:407] Connection header received
0927 10:44:51.574899 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.574903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.574951 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.575034 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:51.575279 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.576019 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.577497 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.577736 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.577815 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.578123 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.578212 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.578265 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.578379 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.578425 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:44:51.578617 12303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.574329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:36987 (local address 127.10.96.193:43751)
0927 10:44:51.574647 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:51.574674 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:51.574796 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:51.575045 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:51.575047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:51.575055 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:51.575180 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.575185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.576312 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.576316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.577315 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:51.577328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.577839 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.577843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:51.577945 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.578002 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:51.578292 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:51.578298 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:44:51.578371 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:44:51.578452 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
I20260927 10:44:51.597898 11741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:51.598063 11741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:51.599982 11741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 238462aeef994682acf920932517873f: Sending DeleteTablet for 3 replicas of tablet e98fc41b6a684ee8b3fb7082bdcdbe45
I20260927 10:44:51.600556 12019 tablet_service.cc:1514] Processing DeleteTablet for tablet e98fc41b6a684ee8b3fb7082bdcdbe45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:51 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:35558
I20260927 10:44:51.600656 12159 tablet_service.cc:1514] Processing DeleteTablet for tablet e98fc41b6a684ee8b3fb7082bdcdbe45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:51 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:46736
I20260927 10:44:51.600757 12271 tablet_replica.cc:330] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: stopping tablet replica
I20260927 10:44:51.600822 12272 tablet_replica.cc:330] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: stopping tablet replica
I20260927 10:44:51.600888 12271 raft_consensus.cc:2229] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:51.600868 11879 tablet_service.cc:1514] Processing DeleteTablet for tablet e98fc41b6a684ee8b3fb7082bdcdbe45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:51 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:36882
I20260927 10:44:51.600978 12272 raft_consensus.cc:2229] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:51.601068 12271 raft_consensus.cc:2258] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:51.601094 12272 raft_consensus.cc:2258] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:51.601222 12273 tablet_replica.cc:330] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: stopping tablet replica
I20260927 10:44:51.601347 12273 raft_consensus.cc:2229] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:51.601496 12273 raft_consensus.cc:2258] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:51.602264 12273 ts_tablet_manager.cc:1905] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:51.602264 12271 ts_tablet_manager.cc:1905] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:51.604931 12271 ts_tablet_manager.cc:1918] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:51.604931 12273 ts_tablet_manager.cc:1918] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:51.605011 12273 log.cc:1196] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/e98fc41b6a684ee8b3fb7082bdcdbe45
I20260927 10:44:51.605011 12271 log.cc:1196] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/e98fc41b6a684ee8b3fb7082bdcdbe45
I20260927 10:44:51.605274 12273 ts_tablet_manager.cc:1939] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 4f27f86c137640608976876a64f6d39e: Deleting consensus metadata
I20260927 10:44:51.605274 12271 ts_tablet_manager.cc:1939] T e98fc41b6a684ee8b3fb7082bdcdbe45 P 19361e3753224eb08824c1ed1c3dc77a: Deleting consensus metadata
I20260927 10:44:51.605808 12272 ts_tablet_manager.cc:1905] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:51.605962 11726 catalog_manager.cc:4841] TS 4f27f86c137640608976876a64f6d39e (127.10.96.193:39329): tablet e98fc41b6a684ee8b3fb7082bdcdbe45 (table test-table [id=dd1c76f8a41042bdafd52b1959af33d5]) successfully deleted
I20260927 10:44:51.606258 11727 catalog_manager.cc:4841] TS 19361e3753224eb08824c1ed1c3dc77a (127.10.96.194:36987): tablet e98fc41b6a684ee8b3fb7082bdcdbe45 (table test-table [id=dd1c76f8a41042bdafd52b1959af33d5]) successfully deleted
I20260927 10:44:51.608564 12272 ts_tablet_manager.cc:1918] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:51.608633 12272 log.cc:1196] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/e98fc41b6a684ee8b3fb7082bdcdbe45
I20260927 10:44:51.608871 12272 ts_tablet_manager.cc:1939] T e98fc41b6a684ee8b3fb7082bdcdbe45 P adbbdca7f04e4186a1372f42425ef0aa: Deleting consensus metadata
I20260927 10:44:51.609592 11726 catalog_manager.cc:4841] TS adbbdca7f04e4186a1372f42425ef0aa (127.10.96.195:36067): tablet e98fc41b6a684ee8b3fb7082bdcdbe45 (table test-table [id=dd1c76f8a41042bdafd52b1959af33d5]) successfully deleted
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[11697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505891, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:44:51.617030 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.606181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.10.96.193:39329)
0927 10:44:51.606297 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.606301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.606867 (+ 566us) server_negotiation.cc:407] Connection header received
0927 10:44:51.607024 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.607027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.607073 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.607159 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:51.607833 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.608353 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.609226 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.609547 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.609645 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.613414 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:51.613423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:51.613434 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:51.613494 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:51.615208 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.615718 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.615723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.615724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.615813 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.616082 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.616084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.616085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.616316 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:51.616473 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.616736 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.616836 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 10:44:51.617530 11939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36894
I20260927 10:44:51.626942 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:51.619266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.10.96.254:43335)
0927 10:44:51.619406 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:51.619412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:51.619430 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:44:51.619548 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:51.619551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:51.619602 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:51.619688 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:51.620361 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.621116 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.621956 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:51.622299 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:51.622419 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:51.623727 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:51.623731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:51.623733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:51.623767 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:51.625329 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.625771 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.625775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.625777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.625870 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:51.626117 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:51.626120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:51.626122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:51.626338 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:51.626484 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:51.626667 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 10:44:51.626783 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 10:44:51.627342 11741 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59082
I20260927 10:44:51.629259 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11814
I20260927 10:44:51.636089 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11954
I20260927 10:44:51.643009 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 12094
I20260927 10:44:51.648341 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 11710
2026-09-27T10:44:51Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2688 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[12311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[12311](info): set up 2 sockets
Sep 27 10:44:51 dist-test-slave-wmg0 krb5kdc[12311](info): commencing operation
krb5kdc: starting...
W20260927 10:44:52.685904 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:44:52 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:44:52.803876 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:32899
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:32899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:52.912401 12324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:52.912717 12324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:52.912794 12324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:52.917184 12324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:52.917294 12324 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:44:52.917344 12324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:52.917403 12324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:52.917443 12324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:52.917498 12324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:52.923091 12324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:32899
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:32899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:52.924224 12324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:52.925261 12324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:44:52.933449 12324 server_base.cc:1034] running on GCE node
W20260927 10:44:52.933386 12330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:52.933346 12332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:52.933351 12329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:52.934155 12324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:52.935254 12324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:52.936442 12324 hybrid_clock.cc:648] HybridClock initialized: now 1790505892936409 us; error 43 us; skew 500 ppm
Sep 27 10:44:52 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505892, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:52.939127 12324 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:44:52.939965 12324 webserver.cc:460] Webserver started at http://127.10.96.254:44337/ using document root <none> and password file <none>
I20260927 10:44:52.940497 12324 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:52.940579 12324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:52.940793 12324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:52.942799 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "eaf51cf0edf540c89bec0fc8c43937ea"
format_stamp: "Formatted at 2026-09-27 10:44:52 on dist-test-slave-wmg0"
server_key: "efeafbf2f5e79e64b348da42a2629cca"
server_key_iv: "3a170da66b72b41d37847e317c0ed215"
server_key_version: "encryptionkey@0"
I20260927 10:44:52.943352 12324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "eaf51cf0edf540c89bec0fc8c43937ea"
format_stamp: "Formatted at 2026-09-27 10:44:52 on dist-test-slave-wmg0"
server_key: "efeafbf2f5e79e64b348da42a2629cca"
server_key_iv: "3a170da66b72b41d37847e317c0ed215"
server_key_version: "encryptionkey@0"
I20260927 10:44:52.946921 12324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:44:52.949278 12339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:52.950484 12324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:52.950610 12324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "eaf51cf0edf540c89bec0fc8c43937ea"
format_stamp: "Formatted at 2026-09-27 10:44:52 on dist-test-slave-wmg0"
server_key: "efeafbf2f5e79e64b348da42a2629cca"
server_key_iv: "3a170da66b72b41d37847e317c0ed215"
server_key_version: "encryptionkey@0"
I20260927 10:44:52.950719 12324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:52.964937 12324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:52.967727 12324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:52.967916 12324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:52.975615 12324 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:32899
I20260927 10:44:52.975647 12401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:32899 every 8 connection(s)
I20260927 10:44:52.976508 12324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:52.979115 12402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:52.979655 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 12324
I20260927 10:44:52.979771 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:44:52.980046 10627 external_mini_cluster.cc:1428] Setting key c5c0d1d8dfcdb44e9962f0688848b6e0
I20260927 10:44:52.985435 12402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Bootstrap starting.
I20260927 10:44:52.988200 12402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:52.989040 12402 log.cc:824] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:44:52 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:52.991053 12402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: No bootstrap required, opened a new log
I20260927 10:44:52.993939 12402 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32899 } }
I20260927 10:44:52.994189 12402 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:52.994272 12402 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf51cf0edf540c89bec0fc8c43937ea, State: Initialized, Role: FOLLOWER
I20260927 10:44:52.994669 12402 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32899 } }
I20260927 10:44:52.994793 12402 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:52.994874 12402 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:52.994952 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:52.981364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.10.96.254:32899)
0927 10:44:52.981798 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:52.981813 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:52.981847 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:44:52.982502 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:52.982518 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:52.982849 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:52.983193 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:52.983962 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:52.984933 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:52.985887 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:52.986503 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:52.986702 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:52.989796 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:52.989802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:52.989819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:52.989850 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:52.992046 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:52.992474 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:52.992480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:52.992490 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:52.992596 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:52.992874 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:52.992879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:52.992880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:52.993270 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:52.993494 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:52.993747 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:44:52.994029 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
I20260927 10:44:52.994966 12402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:52.995875 12402 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32899 } }
I20260927 10:44:52.996236 12402 leader_election.cc:302] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf51cf0edf540c89bec0fc8c43937ea; no voters:
I20260927 10:44:52.996547 12402 leader_election.cc:288] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:52.996647 12407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:52.996863 12407 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [term 1 LEADER]: Becoming Leader. State: Replica: eaf51cf0edf540c89bec0fc8c43937ea, State: Running, Role: LEADER
I20260927 10:44:52.997210 12407 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32899 } }
I20260927 10:44:52.997752 12402 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:52.998999 12409 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf51cf0edf540c89bec0fc8c43937ea. Latest consensus state: current_term: 1 leader_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32899 } } }
I20260927 10:44:52.999203 12409 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:52.999178 12408 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf51cf0edf540c89bec0fc8c43937ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32899 } } }
I20260927 10:44:52.999321 12408 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:52.999790 12419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:53.002431 12419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:53.007982 12419 catalog_manager.cc:1347] Generated new cluster ID: 1dcf281974c540a99c974c4b41feddf6
I20260927 10:44:53.008051 12419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:53.036480 12419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:53.037858 12419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:53.043697 12419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Generated new TSK 0
I20260927 10:44:53.044502 12419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:44:53.080586 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32899
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:53.190145 12428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:53.190392 12428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:53.190438 12428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:53.194357 12428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:53.194406 12428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:53.194474 12428 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:44:53.194504 12428 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:53.194551 12428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:44:53.200030 12428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:32899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:53.201027 12428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:53.202052 12428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:53.211144 12436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:53.211181 12433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:53.211310 12428 server_base.cc:1034] running on GCE node
W20260927 10:44:53.211150 12434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:53.211728 12428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:53.212353 12428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:53.213523 12428 hybrid_clock.cc:648] HybridClock initialized: now 1790505893213483 us; error 46 us; skew 500 ppm
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:53.216084 12428 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:44:53.216838 12428 webserver.cc:460] Webserver started at http://127.10.96.193:37505/ using document root <none> and password file <none>
I20260927 10:44:53.217353 12428 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:53.217409 12428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:53.217626 12428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:53.219564 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "5611c30d0506451286bde974954dfd81"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "659743f3c5097728ef50ebb3294e479b"
server_key_iv: "688b4fac885930eb83468a0ffab90d4a"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.220088 12428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "5611c30d0506451286bde974954dfd81"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "659743f3c5097728ef50ebb3294e479b"
server_key_iv: "688b4fac885930eb83468a0ffab90d4a"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.223665 12428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:44:53.226017 12443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:53.227202 12428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:53.227294 12428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "5611c30d0506451286bde974954dfd81"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "659743f3c5097728ef50ebb3294e479b"
server_key_iv: "688b4fac885930eb83468a0ffab90d4a"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.227380 12428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:53.249665 12428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:53.252574 12428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:53.252731 12428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:53.254235 12428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:53.254285 12428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:53.254325 12428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:53.254375 12428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:53.264865 12428 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:43287
I20260927 10:44:53.266110 12428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:53.266505 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 12428
I20260927 10:44:53.266621 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:53.266918 10627 external_mini_cluster.cc:1428] Setting key 4fbd69d9ef235d02c57ac19903646db1
I20260927 10:44:53.267858 12558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:43287 every 8 connection(s)
I20260927 10:44:53.272589 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.254938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:60641 (local address 127.10.96.254:32899)
0927 10:44:53.255083 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.255090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.255957 (+ 867us) server_negotiation.cc:407] Connection header received
0927 10:44:53.256911 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.256914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.256972 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.257055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.258714 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.259563 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.261434 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.261789 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.261877 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.265435 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.265471 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.265476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.265505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.268905 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.270490 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.270494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.270495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.270595 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.271068 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.271071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.271073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.271327 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.271424 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.272142 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.272346 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:44:53.273342 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.255370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.10.96.193:60641)
0927 10:44:53.255801 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.255850 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.256703 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.257371 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.257379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.257945 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:53.258562 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.258584 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.259703 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.259710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.261300 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.261312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.261945 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.261949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.262183 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.263136 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:53.263192 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:53.265230 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:53.269931 (+ 4701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.269940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.269969 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.270334 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.270732 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.270735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.270737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.270945 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.271464 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:53.271474 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:53.271814 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.272042 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.272390 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":122,"threads_started":1}
I20260927 10:44:53.274571 12559 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32899
I20260927 10:44:53.274699 12559 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:53.275242 12559 heartbeater.cc:511] Master 127.10.96.254:32899 requested a full tablet report, sending...
I20260927 10:44:53.276759 12355 ts_manager.cc:194] Registered new tserver with Master: 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287)
I20260927 10:44:53.278201 12355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:60641
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:44:53.285372 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.276920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:47937 (local address 127.10.96.254:32899)
0927 10:44:53.277025 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.277030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.277544 (+ 514us) server_negotiation.cc:407] Connection header received
0927 10:44:53.277686 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.277689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.277733 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.277814 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.278371 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.279181 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.280370 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.280596 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.280711 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.281902 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.281909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.281915 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.281946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.283559 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.284109 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.284112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.284113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.284193 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.284517 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.284520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.284521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.284686 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.284765 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.285116 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.285259 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:44:53.285486 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.277122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.10.96.193:47937)
0927 10:44:53.277423 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.277441 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.277582 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.277853 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.277859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.278078 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:53.278230 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.278239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.279337 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.279340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.280211 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.280220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.280733 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.280735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.280803 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.281362 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:53.281376 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:53.281765 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:53.283678 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.283681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.283683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.283944 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.284319 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.284325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.284330 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.284429 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.284779 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:53.284785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:53.284896 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.285016 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.285179 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":60,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:44:53.297899 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32899
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:53.408083 12568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:53.408313 12568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:53.408396 12568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:53.412324 12568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:53.412379 12568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:53.412416 12568 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:44:53.412470 12568 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:53.412537 12568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:44:53.417676 12568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:32899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:53.418632 12568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:53.419632 12568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:53.428835 12573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:53.428984 12568 server_base.cc:1034] running on GCE node
W20260927 10:44:53.428824 12576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:53.428815 12574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:53.429543 12568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:53.430191 12568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:53.431373 12568 hybrid_clock.cc:648] HybridClock initialized: now 1790505893431360 us; error 34 us; skew 500 ppm
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:53.433928 12568 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:44:53.434685 12568 webserver.cc:460] Webserver started at http://127.10.96.194:33393/ using document root <none> and password file <none>
I20260927 10:44:53.435170 12568 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:53.435228 12568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:53.435384 12568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:53.437276 12568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "6926622074dc441dba9d187fd3bfa8f2"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "743723a00648d6f28bfd6adaf3c0f229"
server_key_iv: "39071d33680556aabc29d70c3b793fc1"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.437848 12568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "6926622074dc441dba9d187fd3bfa8f2"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "743723a00648d6f28bfd6adaf3c0f229"
server_key_iv: "39071d33680556aabc29d70c3b793fc1"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.441226 12568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:53.443573 12583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:53.444725 12568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:53.444835 12568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "6926622074dc441dba9d187fd3bfa8f2"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "743723a00648d6f28bfd6adaf3c0f229"
server_key_iv: "39071d33680556aabc29d70c3b793fc1"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.444929 12568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:53.457790 12568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:53.460865 12568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:53.461048 12568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:53.462582 12568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:53.462651 12568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:53.462745 12568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:53.462790 12568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:53.473068 12698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:34539 every 8 connection(s)
I20260927 10:44:53.473047 12568 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:34539
I20260927 10:44:53.474080 12568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:53.477242 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.463364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:37773 (local address 127.10.96.254:32899)
0927 10:44:53.463503 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.463509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.464379 (+ 870us) server_negotiation.cc:407] Connection header received
0927 10:44:53.465499 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.465502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.465557 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.465653 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.467170 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.468063 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.469827 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.470200 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.470302 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.473482 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.473486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.473489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.473517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.475198 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.475738 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.475741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.475742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.475835 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.476169 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.476172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.476174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.476362 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.476462 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.476943 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.477061 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:44:53.478034 12592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.463786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.10.96.194:37773)
0927 10:44:53.464222 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.464270 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.465262 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.465836 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.465845 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.466347 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:53.466996 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.467018 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.468211 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.468215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.469703 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.469713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.470405 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.470410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.470631 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.471428 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:53.471452 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:53.473303 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:53.475320 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.475324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.475335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.475646 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.475934 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.475937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.475939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.476075 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.476471 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:53.476480 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:53.476711 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.476887 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.477203 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":121,"threads_started":1}
I20260927 10:44:53.479074 12699 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32899
I20260927 10:44:53.479231 12699 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:53.479696 12699 heartbeater.cc:511] Master 127.10.96.254:32899 requested a full tablet report, sending...
I20260927 10:44:53.480669 12356 ts_manager.cc:194] Registered new tserver with Master: 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539)
I20260927 10:44:53.481458 12356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:37773
I20260927 10:44:53.483996 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 12568
I20260927 10:44:53.484119 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:53.484452 10627 external_mini_cluster.cc:1428] Setting key 5e1d098a2c62fcd8a1d740f0d9ead803
I20260927 10:44:53.489826 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.481154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.10.96.194:59493)
0927 10:44:53.481551 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.481565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.481654 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.481930 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.481933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.482125 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:53.482223 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.482228 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.483161 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.483163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.483721 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.483728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.484179 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.484183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.484284 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.484759 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:53.484772 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:53.485146 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:53.488307 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.488309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.488311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.488492 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.488780 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.488783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.488784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.488851 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.489349 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:53.489351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:53.489418 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.489542 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.489702 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":95,"threads_started":1}
I20260927 10:44:53.489948 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.481259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:59493 (local address 127.10.96.254:32899)
0927 10:44:53.481379 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.481383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.481623 (+ 240us) server_negotiation.cc:407] Connection header received
0927 10:44:53.481746 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.481752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.481828 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.481906 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.482329 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.483053 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.483856 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.484065 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.484175 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.485286 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.485289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.485292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.485311 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.488203 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.488589 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.488595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.488600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.488690 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.488935 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.488941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.488946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.489259 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.489367 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.489608 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.489738 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:44:53.510480 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32899
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:42033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:53.620555 12708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:53.620831 12708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:53.620882 12708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:53.624996 12708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:53.625093 12708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:53.625177 12708 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 10:44:53.625209 12708 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:44:53.625267 12708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:44:53.630621 12708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:32899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:53.631701 12708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:53.632721 12708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:53.641965 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:53.641925 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:53.641947 12708 server_base.cc:1034] running on GCE node
W20260927 10:44:53.641978 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:53.642585 12708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:53.643148 12708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:53.644309 12708 hybrid_clock.cc:648] HybridClock initialized: now 1790505893644292 us; error 33 us; skew 500 ppm
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:53.647143 12708 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:44:53.648083 12708 webserver.cc:460] Webserver started at http://127.10.96.195:45901/ using document root <none> and password file <none>
I20260927 10:44:53.648670 12708 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:53.648756 12708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:53.648960 12708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:53.651006 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "6ac98db4b6514695902e473e5551d96c"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "84dd163bb402bc166eaa4239e9fc90c0"
server_key_iv: "e8f22faf1ad5fc87d25405efaac2615d"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.651520 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "6ac98db4b6514695902e473e5551d96c"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "84dd163bb402bc166eaa4239e9fc90c0"
server_key_iv: "e8f22faf1ad5fc87d25405efaac2615d"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.655272 12708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 10:44:53.657665 12723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:53.658905 12708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:44:53.659003 12708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "6ac98db4b6514695902e473e5551d96c"
format_stamp: "Formatted at 2026-09-27 10:44:53 on dist-test-slave-wmg0"
server_key: "84dd163bb402bc166eaa4239e9fc90c0"
server_key_iv: "e8f22faf1ad5fc87d25405efaac2615d"
server_key_version: "encryptionkey@0"
I20260927 10:44:53.659086 12708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:53.695935 12708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:53.699265 12708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:53.699477 12708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:53.700935 12708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:53.700982 12708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:53.701021 12708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:53.701074 12708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:53.711910 12708 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:35571
I20260927 10:44:53.712092 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:35571 every 8 connection(s)
I20260927 10:44:53.713061 12708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:44:53.715150 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.701869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:46991 (local address 127.10.96.254:32899)
0927 10:44:53.701984 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.701989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.703138 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 10:44:53.704273 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.704277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.704338 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.704440 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.706151 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.707041 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.708253 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.708495 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.708609 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.711234 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.711240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.711246 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.711281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.713031 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.713665 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.713668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.713670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.713765 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.714137 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.714140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.714142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.714378 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.714458 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.714931 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.715032 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:44:53.715979 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.702456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.10.96.195:46991)
0927 10:44:53.702976 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.703018 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.704047 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.704665 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.704675 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.705170 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:53.705971 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.705998 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.707162 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.707165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.708135 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.708144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.708664 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.708669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.708876 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.709514 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:53.709532 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:53.711067 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:53.713181 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.713188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.713203 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.713537 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.713863 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.713866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.713868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.714038 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.714485 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:53.714490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:53.714714 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.714863 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.715129 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":162,"threads_started":1}
I20260927 10:44:53.716650 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 12708
I20260927 10:44:53.716765 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:44:53.717064 10627 external_mini_cluster.cc:1428] Setting key aef73c119e28963c44806813c3d6baea
I20260927 10:44:53.717346 12839 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32899
I20260927 10:44:53.717542 12839 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:53.718142 12839 heartbeater.cc:511] Master 127.10.96.254:32899 requested a full tablet report, sending...
I20260927 10:44:53.719267 12356 ts_manager.cc:194] Registered new tserver with Master: 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:53.720072 12356 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:46991
I20260927 10:44:53.726266 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.719641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:37341 (local address 127.10.96.254:32899)
0927 10:44:53.719745 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.719748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.719947 (+ 199us) server_negotiation.cc:407] Connection header received
0927 10:44:53.720052 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.720054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.720126 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.720233 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.720801 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.721302 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.722144 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.722373 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.722478 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.723461 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.723464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.723467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.723491 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.724696 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.725096 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.725101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.725103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.725182 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.725447 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.725452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.725454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.725642 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.725771 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.726018 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.726121 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:44:53.726285 12845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.719590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.10.96.195:37341)
0927 10:44:53.719871 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.719886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.719974 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.720346 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.720349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.720570 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:53.720693 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.720700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.721418 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.721421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.722033 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.722041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.722478 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.722481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.722570 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.723008 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:53.723019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:53.723338 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:53.724814 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.724817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.724819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.725005 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.725271 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:53.725275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:53.725277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:53.725356 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:53.725761 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:53.725764 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:53.725857 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.725969 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.726109 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260927 10:44:53.730724 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:44:53 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:53.756060 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.747793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.10.96.254:32899)
0927 10:44:53.747943 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.747948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.747969 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:44:53.748013 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.748015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.748056 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.748133 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:53.748848 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.749422 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.750264 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.750467 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.750545 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.752973 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:53.752976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:53.752978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:53.753002 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:53.754451 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.754865 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.754871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.754873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.754953 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:53.755211 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:53.755216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:53.755218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:53.755392 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:53.755526 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.755770 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.755898 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:44:53.759007 12356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:53.762094 12356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:53.777009 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.771197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.10.96.195:35571)
0927 10:44:53.771497 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.771511 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.771780 (+ 269us) server_negotiation.cc:407] Connection header received
0927 10:44:53.771925 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.771934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.772164 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.772329 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:53.772752 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.774442 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.775822 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.776426 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.776536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.776685 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.776757 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.776859 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260927 10:44:53.777050 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.770399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.0.0.1:40536)
0927 10:44:53.770783 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.770995 (+ 212us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.771135 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.772657 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.772660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.772680 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:53.772786 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.772790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.773968 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.773975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.775544 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.775557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.776592 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.776595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.776728 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.776749 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.776825 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.776885 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":126,"thread_start_us":91,"threads_started":1}
I20260927 10:44:53.776945 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.770823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35571 (local address 127.0.0.1:52238)
0927 10:44:53.771667 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.771681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.771789 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.772349 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.772357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.772386 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:53.772616 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.772624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.774537 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.774541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.775723 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.775729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.776546 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.776551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.776645 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.776668 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.776717 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.776782 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":58,"thread_start_us":93,"threads_started":1}
I20260927 10:44:53.777386 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.770514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.10.96.194:34539)
0927 10:44:53.772222 (+ 1708us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.772238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.772262 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:44:53.772332 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.772337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.772499 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.772898 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:53.773001 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.773811 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.775655 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.776473 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.776666 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.776842 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.776890 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.776981 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"thread_start_us":129,"threads_started":1}
I20260927 10:44:53.777904 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.771111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43287 (local address 127.0.0.1:58352)
0927 10:44:53.771874 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:53.771887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:53.771979 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:53.772407 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:53.772410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:53.772422 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:53.772644 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.772652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.774024 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.774026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.775256 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:53.775265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.776286 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.776290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:53.776395 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.776436 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:44:53.776536 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:44:53.777793 (+ 1257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":163,"thread_start_us":114,"threads_started":1}
I20260927 10:44:53.778357 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:53.771529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.10.96.193:43287)
0927 10:44:53.771794 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:53.771809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:53.771973 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:44:53.772092 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:53.772097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:53.772253 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:53.772575 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:53.772745 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.773900 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.775375 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:53.776162 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:53.776285 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:53.776430 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:53.778172 (+ 1742us) server_negotiation.cc:299] Negotiation successful
0927 10:44:53.778262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260927 10:44:53.780324 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 33efaa0c6e764c8ab61d64a22e766567 (DEFAULT_TABLE table=test-table [id=82dd0b451a9b412a97871e1c21abfdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:53.780324 12633 tablet_service.cc:1467] Processing CreateTablet for tablet 33efaa0c6e764c8ab61d64a22e766567 (DEFAULT_TABLE table=test-table [id=82dd0b451a9b412a97871e1c21abfdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:53.780732 12493 tablet_service.cc:1467] Processing CreateTablet for tablet 33efaa0c6e764c8ab61d64a22e766567 (DEFAULT_TABLE table=test-table [id=82dd0b451a9b412a97871e1c21abfdf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:53.781453 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33efaa0c6e764c8ab61d64a22e766567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:53.781447 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33efaa0c6e764c8ab61d64a22e766567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:53.781723 12493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33efaa0c6e764c8ab61d64a22e766567. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:53.787434 12862 tablet_bootstrap.cc:492] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Bootstrap starting.
I20260927 10:44:53.789007 12863 tablet_bootstrap.cc:492] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Bootstrap starting.
I20260927 10:44:53.790100 12862 tablet_bootstrap.cc:654] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:53.791298 12863 tablet_bootstrap.cc:654] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:53.791495 12862 log.cc:824] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:53.792138 12863 log.cc:824] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:53.792196 12864 tablet_bootstrap.cc:492] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Bootstrap starting.
I20260927 10:44:53.793421 12862 tablet_bootstrap.cc:492] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: No bootstrap required, opened a new log
I20260927 10:44:53.793648 12862 ts_tablet_manager.cc:1397] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 10:44:53.794134 12863 tablet_bootstrap.cc:492] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: No bootstrap required, opened a new log
I20260927 10:44:53.794232 12864 tablet_bootstrap.cc:654] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:53.794328 12863 ts_tablet_manager.cc:1397] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260927 10:44:53.795022 12864 log.cc:824] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:53.796967 12864 tablet_bootstrap.cc:492] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: No bootstrap required, opened a new log
I20260927 10:44:53.797154 12864 ts_tablet_manager.cc:1397] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:44:53.797168 12862 raft_consensus.cc:348] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:53.797439 12862 raft_consensus.cc:374] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:53.797508 12862 raft_consensus.cc:729] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5611c30d0506451286bde974954dfd81, State: Initialized, Role: FOLLOWER
I20260927 10:44:53.797948 12862 consensus_queue.cc:260] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:53.798170 12863 raft_consensus.cc:348] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:53.798414 12863 raft_consensus.cc:374] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:53.798472 12863 raft_consensus.cc:729] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac98db4b6514695902e473e5551d96c, State: Initialized, Role: FOLLOWER
I20260927 10:44:53.798636 12559 heartbeater.cc:503] Master 127.10.96.254:32899 was elected leader, sending a full tablet report...
I20260927 10:44:53.798959 12863 consensus_queue.cc:260] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:53.799569 12839 heartbeater.cc:503] Master 127.10.96.254:32899 was elected leader, sending a full tablet report...
I20260927 10:44:53.799921 12862 ts_tablet_manager.cc:1428] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 10:44:53.799921 12863 ts_tablet_manager.cc:1428] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 10:44:53.802170 12864 raft_consensus.cc:348] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:53.802418 12864 raft_consensus.cc:374] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:53.802479 12864 raft_consensus.cc:729] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Initialized, Role: FOLLOWER
I20260927 10:44:53.802881 12864 consensus_queue.cc:260] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:53.803895 12864 ts_tablet_manager.cc:1428] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:44:53.803905 12699 heartbeater.cc:503] Master 127.10.96.254:32899 was elected leader, sending a full tablet report...
W20260927 10:44:53.964079 12840 tablet.cc:2369] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:44:53.975208 12700 tablet.cc:2369] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:44:54.018345 12560 tablet.cc:2369] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:44:54.133898 12870 raft_consensus.cc:482] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:54.134167 12870 raft_consensus.cc:504] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.135625 12870 leader_election.cc:288] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:54.139495 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.135943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:58163 (local address 127.10.96.193:43287)
0927 10:44:54.136125 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:54.136132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:54.136198 (+ 66us) server_negotiation.cc:407] Connection header received
0927 10:44:54.136376 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:54.136379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:54.136419 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:54.136499 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:54.136760 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.137297 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.138527 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.138929 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.139052 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.139104 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:54.139299 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:44:54.139371 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 10:44:54.139496 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.136036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:55495 (local address 127.10.96.195:35571)
0927 10:44:54.136160 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:54.136165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:54.136351 (+ 186us) server_negotiation.cc:407] Connection header received
0927 10:44:54.136496 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:54.136503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:54.136544 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:54.136624 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:54.136880 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.137385 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.138518 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.138910 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.139032 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.139087 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:54.139280 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 10:44:54.139358 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:44:54.139504 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.135807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43287 (local address 127.10.96.194:58163)
0927 10:44:54.136122 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:54.136136 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:54.136256 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:54.136510 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:54.136513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:54.136528 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:54.136638 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.136645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.137402 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.137406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.138402 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.138411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.139041 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.139044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.139166 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.139191 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:44:54.139284 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:44:54.139371 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":62,"thread_start_us":106,"threads_started":1}
I20260927 10:44:54.139516 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.135913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35571 (local address 127.10.96.194:55495)
0927 10:44:54.136262 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:54.136273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:54.136387 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:54.136633 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:54.136636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:54.136651 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:54.136773 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.136779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.137520 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.137524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.138396 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.138404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.139025 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.139028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.139120 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.139180 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 10:44:54.139265 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:44:54.139365 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":73,"thread_start_us":77,"threads_started":1}
I20260927 10:44:54.140185 12793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33efaa0c6e764c8ab61d64a22e766567" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c" is_pre_election: true
I20260927 10:44:54.140185 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33efaa0c6e764c8ab61d64a22e766567" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81" is_pre_election: true
I20260927 10:44:54.140494 12793 raft_consensus.cc:2454] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 0.
I20260927 10:44:54.140506 12513 raft_consensus.cc:2454] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 0.
I20260927 10:44:54.141011 12584 leader_election.cc:302] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2, 6ac98db4b6514695902e473e5551d96c; no voters:
I20260927 10:44:54.141291 12870 raft_consensus.cc:2781] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:54.141364 12870 raft_consensus.cc:482] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:54.141412 12870 raft_consensus.cc:3037] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.142508 12870 raft_consensus.cc:504] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.142872 12870 leader_election.cc:288] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:54.143298 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33efaa0c6e764c8ab61d64a22e766567" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81"
I20260927 10:44:54.143397 12793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33efaa0c6e764c8ab61d64a22e766567" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c"
I20260927 10:44:54.143435 12513 raft_consensus.cc:3037] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.143532 12793 raft_consensus.cc:3037] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.144815 12513 raft_consensus.cc:2454] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 1.
I20260927 10:44:54.145170 12587 leader_election.cc:302] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:54.145288 12793 raft_consensus.cc:2454] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 1.
I20260927 10:44:54.145368 12870 raft_consensus.cc:2781] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:54.145587 12870 raft_consensus.cc:686] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Running, Role: LEADER
I20260927 10:44:54.145967 12870 consensus_queue.cc:237] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.149130 12352 catalog_manager.cc:5495] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194). New cstate: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:44:54 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505892, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:54.255628 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.247219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32899 (local address 127.0.0.1:54756)
0927 10:44:54.247517 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:54.247534 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:54.247647 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:54.247923 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:54.247926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:54.248387 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:54.248484 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.248490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.249331 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.249336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.249907 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.249915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.250334 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.250337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.250418 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.250947 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:54.250958 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:54.252304 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:54.254111 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:54.254119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:54.254124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:54.254381 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:54.254617 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:54.254619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:54.254621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:54.254697 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:54.255043 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:54.255049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:54.255226 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 10:44:54.255339 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:44:54.255461 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":115,"threads_started":1}
I20260927 10:44:54.255632 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.247303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.10.96.254:32899)
0927 10:44:54.247447 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:54.247451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:54.247642 (+ 191us) server_negotiation.cc:407] Connection header received
0927 10:44:54.247717 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:54.247719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:54.247762 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:54.247890 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:54.248602 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.249201 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.250020 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.250225 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.250332 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.252431 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:54.252435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:54.252437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:54.252462 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:54.253969 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:54.254467 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:54.254470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:54.254471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:54.254542 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:54.254770 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:54.254772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:54.254774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:54.254928 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:54.255001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:54.255392 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 10:44:54.255515 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":64}
I20260927 10:44:54.259459 12352 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54756:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:44:54.260106 12352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:54.265069 12633 tablet_service.cc:1467] Processing CreateTablet for tablet 93312172ccd744ab9c33c59f80f067d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a8ec97eb5ee44508974b5807eb6bb95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:54.265023 12493 tablet_service.cc:1467] Processing CreateTablet for tablet 93312172ccd744ab9c33c59f80f067d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a8ec97eb5ee44508974b5807eb6bb95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:54.265836 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93312172ccd744ab9c33c59f80f067d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:54.265836 12493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93312172ccd744ab9c33c59f80f067d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:54.268260 12864 tablet_bootstrap.cc:492] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: Bootstrap starting.
I20260927 10:44:54.268927 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 93312172ccd744ab9c33c59f80f067d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a8ec97eb5ee44508974b5807eb6bb95f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:44:54.269644 12864 tablet_bootstrap.cc:654] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:54.269640 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93312172ccd744ab9c33c59f80f067d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:54.269639 12862 tablet_bootstrap.cc:492] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: Bootstrap starting.
I20260927 10:44:54.270784 12862 tablet_bootstrap.cc:654] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:54.270829 12864 tablet_bootstrap.cc:492] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: No bootstrap required, opened a new log
I20260927 10:44:54.270902 12864 ts_tablet_manager.cc:1397] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:44:54.271376 12864 raft_consensus.cc:348] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.271490 12864 raft_consensus.cc:374] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:54.271526 12864 raft_consensus.cc:729] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Initialized, Role: FOLLOWER
I20260927 10:44:54.271648 12864 consensus_queue.cc:260] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.271910 12864 ts_tablet_manager.cc:1428] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:44:54.272107 12862 tablet_bootstrap.cc:492] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: No bootstrap required, opened a new log
I20260927 10:44:54.272176 12862 ts_tablet_manager.cc:1397] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:44:54.272604 12862 raft_consensus.cc:348] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.272656 12863 tablet_bootstrap.cc:492] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: Bootstrap starting.
I20260927 10:44:54.272699 12862 raft_consensus.cc:374] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:54.272732 12862 raft_consensus.cc:729] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5611c30d0506451286bde974954dfd81, State: Initialized, Role: FOLLOWER
I20260927 10:44:54.273213 12870 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.273345 12870 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.272841 12862 consensus_queue.cc:260] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.273593 12863 tablet_bootstrap.cc:654] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:54.273766 12862 ts_tablet_manager.cc:1428] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:54.273929 12868 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.274030 12868 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.274837 12863 tablet_bootstrap.cc:492] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: No bootstrap required, opened a new log
I20260927 10:44:54.274932 12863 ts_tablet_manager.cc:1397] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:44:54.275291 12863 raft_consensus.cc:348] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.275394 12863 raft_consensus.cc:374] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:54.275445 12863 raft_consensus.cc:729] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac98db4b6514695902e473e5551d96c, State: Initialized, Role: FOLLOWER
I20260927 10:44:54.275584 12863 consensus_queue.cc:260] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.275844 12863 ts_tablet_manager.cc:1428] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:44:54.275832 12869 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.275996 12869 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.347239 12870 raft_consensus.cc:482] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:54.347409 12870 raft_consensus.cc:504] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.347892 12870 leader_election.cc:288] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:54.348333 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93312172ccd744ab9c33c59f80f067d5" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81" is_pre_election: true
I20260927 10:44:54.348400 12793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93312172ccd744ab9c33c59f80f067d5" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c" is_pre_election: true
I20260927 10:44:54.348503 12513 raft_consensus.cc:2454] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 0.
I20260927 10:44:54.348598 12793 raft_consensus.cc:2454] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 0.
I20260927 10:44:54.348788 12587 leader_election.cc:302] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:54.348968 12870 raft_consensus.cc:2781] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:54.349033 12870 raft_consensus.cc:482] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:54.349062 12870 raft_consensus.cc:3037] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.350090 12870 raft_consensus.cc:504] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.350417 12870 leader_election.cc:288] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:54.350718 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93312172ccd744ab9c33c59f80f067d5" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81"
I20260927 10:44:54.350756 12793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93312172ccd744ab9c33c59f80f067d5" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c"
I20260927 10:44:54.350847 12513 raft_consensus.cc:3037] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.350868 12793 raft_consensus.cc:3037] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.351711 12793 raft_consensus.cc:2454] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 1.
I20260927 10:44:54.351706 12513 raft_consensus.cc:2454] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 1.
I20260927 10:44:54.351979 12587 leader_election.cc:302] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:54.352182 12870 raft_consensus.cc:2781] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:54.352281 12870 raft_consensus.cc:686] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Running, Role: LEADER
I20260927 10:44:54.352428 12870 consensus_queue.cc:237] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.353041 12873 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: TxnStatusTablet state changed. Reason: New leader 6926622074dc441dba9d187fd3bfa8f2. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.353152 12873 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:54.353425 12885 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:44:54.353720 12885 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:44:54.354084 12355 catalog_manager.cc:5495] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194). New cstate: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } health_report { overall_health: UNKNOWN } } }
I20260927 10:44:54.377408 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.373143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.0.0.1:40538)
0927 10:44:54.373292 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:54.373311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:54.373418 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:54.374090 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:54.374093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:54.374101 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:54.374232 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.374239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.375089 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.375094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.375766 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.375774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.376292 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.376295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.376384 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.376433 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:54.377099 (+ 666us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:54.377123 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:44:54.377194 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:44:54.377270 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:44:54.377430 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.373501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.10.96.194:34539)
0927 10:44:54.373780 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:54.373788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:54.373802 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:54.373872 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:54.373875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:54.373924 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:54.374065 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:54.374350 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.374985 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.375878 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.376186 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.376294 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.376602 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:54.377002 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:54.377136 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:54.377207 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:44:54.377258 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260927 10:44:54.382553 12793 raft_consensus.cc:1261] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Refusing update from remote peer 6926622074dc441dba9d187fd3bfa8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:54.382673 12513 raft_consensus.cc:1261] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Refusing update from remote peer 6926622074dc441dba9d187fd3bfa8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:54.383311 12873 consensus_queue.cc:1035] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:54.383509 12870 consensus_queue.cc:1035] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:54.387831 12868 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: TxnStatusTablet state changed. Reason: New leader 6926622074dc441dba9d187fd3bfa8f2. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.387945 12868 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.388406 12869 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: TxnStatusTablet state changed. Reason: New leader 6926622074dc441dba9d187fd3bfa8f2. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.388540 12869 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.391698 12870 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.391829 12870 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:54.392017 12891 mvcc.cc:205] Tried to move back new op lower bound from 7333912143384412160 to 7333912143268671488. Current Snapshot: MvccSnapshot[applied={T|T < 7333912143384412160}]
I20260927 10:44:54.392169 12868 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.392272 12868 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 5611c30d0506451286bde974954dfd81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.392444 12889 mvcc.cc:205] Tried to move back new op lower bound from 7333912143384412160 to 7333912143268671488. Current Snapshot: MvccSnapshot[applied={T|T < 7333912143384412160}]
I20260927 10:44:54.392630 12870 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.392721 12870 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6926622074dc441dba9d187fd3bfa8f2: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:44:54.394284 12869 tablet_replica.cc:439] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } } }
I20260927 10:44:54.394394 12869 tablet_replica.cc:442] T 93312172ccd744ab9c33c59f80f067d5 P 6ac98db4b6514695902e473e5551d96c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:44:54.395391 12890 mvcc.cc:205] Tried to move back new op lower bound from 7333912143384412160 to 7333912143268671488. Current Snapshot: MvccSnapshot[applied={T|T < 7333912143384412160 or (T in {7333912143384412160})}]
I20260927 10:44:54.401782 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.398008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40546 (local address 127.10.96.194:34539)
0927 10:44:54.398114 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:54.398121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:54.398341 (+ 220us) server_negotiation.cc:407] Connection header received
0927 10:44:54.398555 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:54.398560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:54.398609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:54.398683 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:54.398931 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.399580 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.400812 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.401027 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.401100 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.401354 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:54.401413 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:54.401514 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:54.401611 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 10:44:54.401655 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 10:44:54.411883 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.408378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:47943 (local address 127.10.96.194:34539)
0927 10:44:54.408522 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:54.408526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:54.408631 (+ 105us) server_negotiation.cc:407] Connection header received
0927 10:44:54.408735 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:54.408737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:54.408772 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:54.408831 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:54.409111 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.409684 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.410905 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.411101 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.411209 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.411514 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:54.411569 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:54.411622 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:54.411735 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 10:44:54.411772 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":96}
I20260927 10:44:54.411989 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:54.408265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.10.96.194:47943)
0927 10:44:54.408546 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:54.408561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:54.408660 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:54.408862 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:54.408865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:54.408872 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:54.409009 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.409015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.409781 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.409787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.410781 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:54.410791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:54.411224 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:54.411228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:54.411328 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:54.411375 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:54.411649 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:54.411655 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:44:54.411721 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:44:54.411795 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260927 10:44:54.417860 12793 raft_consensus.cc:1261] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Refusing update from remote peer 6926622074dc441dba9d187fd3bfa8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:54.417912 12513 raft_consensus.cc:1261] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Refusing update from remote peer 6926622074dc441dba9d187fd3bfa8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:54.418273 12894 consensus_queue.cc:1035] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:54.418664 12894 consensus_queue.cc:1035] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:54.457166 12356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:54.457352 12356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:54.459789 12356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Sending DeleteTablet for 3 replicas of tablet 33efaa0c6e764c8ab61d64a22e766567
I20260927 10:44:54.460407 12633 tablet_service.cc:1514] Processing DeleteTablet for tablet 33efaa0c6e764c8ab61d64a22e766567 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:54 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:40536
I20260927 10:44:54.460546 12493 tablet_service.cc:1514] Processing DeleteTablet for tablet 33efaa0c6e764c8ab61d64a22e766567 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:54 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:58352
I20260927 10:44:54.460644 12773 tablet_service.cc:1514] Processing DeleteTablet for tablet 33efaa0c6e764c8ab61d64a22e766567 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:54 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:52238
I20260927 10:44:54.461059 12908 tablet_replica.cc:330] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: stopping tablet replica
I20260927 10:44:54.461079 12906 tablet_replica.cc:330] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: stopping tablet replica
I20260927 10:44:54.461393 12908 raft_consensus.cc:2229] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:54.461393 12906 raft_consensus.cc:2229] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:54.461625 12908 raft_consensus.cc:2258] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:54.461647 12356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:44:54.461920 12906 raft_consensus.cc:2258] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:44:54.462054 12356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:54.463052 12906 ts_tablet_manager.cc:1905] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:54.463052 12908 ts_tablet_manager.cc:1905] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:54.465961 12908 ts_tablet_manager.cc:1918] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:54.466001 12906 ts_tablet_manager.cc:1918] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:54.466063 12908 log.cc:1196] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/33efaa0c6e764c8ab61d64a22e766567
I20260927 10:44:54.466141 12906 log.cc:1196] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/33efaa0c6e764c8ab61d64a22e766567
I20260927 10:44:54.466514 12906 ts_tablet_manager.cc:1939] T 33efaa0c6e764c8ab61d64a22e766567 P 6926622074dc441dba9d187fd3bfa8f2: Deleting consensus metadata
I20260927 10:44:54.466506 12908 ts_tablet_manager.cc:1939] T 33efaa0c6e764c8ab61d64a22e766567 P 5611c30d0506451286bde974954dfd81: Deleting consensus metadata
I20260927 10:44:54.467599 12343 catalog_manager.cc:4841] TS 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287): tablet 33efaa0c6e764c8ab61d64a22e766567 (table test-table [id=82dd0b451a9b412a97871e1c21abfdf9]) successfully deleted
I20260927 10:44:54.467856 12907 tablet_replica.cc:330] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: stopping tablet replica
I20260927 10:44:54.467867 12493 tablet_service.cc:1467] Processing CreateTablet for tablet 74ff2fd698694215a622445132724ebe (DEFAULT_TABLE table=test-table [id=c494ce9efd9d4419b234df1356c89bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:54.468103 12493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ff2fd698694215a622445132724ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:54.468160 12907 raft_consensus.cc:2229] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:54.468245 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 74ff2fd698694215a622445132724ebe (DEFAULT_TABLE table=test-table [id=c494ce9efd9d4419b234df1356c89bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:54.468346 12907 raft_consensus.cc:2258] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:54.468487 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ff2fd698694215a622445132724ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:54.469595 12907 ts_tablet_manager.cc:1905] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:54.470359 12862 tablet_bootstrap.cc:492] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Bootstrap starting.
I20260927 10:44:54.470844 12343 catalog_manager.cc:4841] TS 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539): tablet 33efaa0c6e764c8ab61d64a22e766567 (table test-table [id=82dd0b451a9b412a97871e1c21abfdf9]) successfully deleted
I20260927 10:44:54.471306 12862 tablet_bootstrap.cc:654] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:54.472477 12907 ts_tablet_manager.cc:1918] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:54.472561 12907 log.cc:1196] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/33efaa0c6e764c8ab61d64a22e766567
I20260927 10:44:54.472635 12633 tablet_service.cc:1467] Processing CreateTablet for tablet 74ff2fd698694215a622445132724ebe (DEFAULT_TABLE table=test-table [id=c494ce9efd9d4419b234df1356c89bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:54.472711 12862 tablet_bootstrap.cc:492] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: No bootstrap required, opened a new log
I20260927 10:44:54.472784 12862 ts_tablet_manager.cc:1397] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:44:54.472872 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ff2fd698694215a622445132724ebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:54.473199 12862 raft_consensus.cc:348] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.473311 12862 raft_consensus.cc:374] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:54.473342 12862 raft_consensus.cc:729] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5611c30d0506451286bde974954dfd81, State: Initialized, Role: FOLLOWER
I20260927 10:44:54.473608 12863 tablet_bootstrap.cc:492] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Bootstrap starting.
I20260927 10:44:54.473450 12862 consensus_queue.cc:260] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.473960 12862 ts_tablet_manager.cc:1428] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:44:54.474599 12863 tablet_bootstrap.cc:654] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:54.477183 12863 tablet_bootstrap.cc:492] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: No bootstrap required, opened a new log
I20260927 10:44:54.477268 12863 ts_tablet_manager.cc:1397] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 10:44:54.477727 12863 raft_consensus.cc:348] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.477841 12863 raft_consensus.cc:374] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:54.477877 12863 raft_consensus.cc:729] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac98db4b6514695902e473e5551d96c, State: Initialized, Role: FOLLOWER
I20260927 10:44:54.477998 12863 consensus_queue.cc:260] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.478236 12863 ts_tablet_manager.cc:1428] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:54.480069 12864 tablet_bootstrap.cc:492] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Bootstrap starting.
I20260927 10:44:54.480991 12864 tablet_bootstrap.cc:654] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:54.486289 12864 tablet_bootstrap.cc:492] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: No bootstrap required, opened a new log
I20260927 10:44:54.486369 12864 ts_tablet_manager.cc:1397] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 10:44:54.486778 12864 raft_consensus.cc:348] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.486873 12864 raft_consensus.cc:374] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:54.486900 12864 raft_consensus.cc:729] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Initialized, Role: FOLLOWER
I20260927 10:44:54.487010 12864 consensus_queue.cc:260] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.487242 12864 ts_tablet_manager.cc:1428] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:44:54.489675 12907 ts_tablet_manager.cc:1939] T 33efaa0c6e764c8ab61d64a22e766567 P 6ac98db4b6514695902e473e5551d96c: Deleting consensus metadata
I20260927 10:44:54.490830 12340 catalog_manager.cc:4841] TS 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571): tablet 33efaa0c6e764c8ab61d64a22e766567 (table test-table [id=82dd0b451a9b412a97871e1c21abfdf9]) successfully deleted
I20260927 10:44:54.615758 12894 raft_consensus.cc:482] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:54.615914 12894 raft_consensus.cc:504] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.616364 12894 leader_election.cc:288] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:54.616905 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ff2fd698694215a622445132724ebe" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c" is_pre_election: true
I20260927 10:44:54.616884 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ff2fd698694215a622445132724ebe" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81" is_pre_election: true
I20260927 10:44:54.617053 12792 raft_consensus.cc:2454] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 0.
I20260927 10:44:54.617048 12513 raft_consensus.cc:2454] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 0.
I20260927 10:44:54.617395 12587 leader_election.cc:302] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:54.617624 12894 raft_consensus.cc:2781] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:54.617700 12894 raft_consensus.cc:482] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:54.617738 12894 raft_consensus.cc:3037] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.619019 12894 raft_consensus.cc:504] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.619393 12894 leader_election.cc:288] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 election: Requested vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:54.619756 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ff2fd698694215a622445132724ebe" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81"
I20260927 10:44:54.619927 12513 raft_consensus.cc:3037] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.619907 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ff2fd698694215a622445132724ebe" candidate_uuid: "6926622074dc441dba9d187fd3bfa8f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c"
I20260927 10:44:54.620019 12792 raft_consensus.cc:3037] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:54.621090 12792 raft_consensus.cc:2454] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 1.
I20260927 10:44:54.621091 12513 raft_consensus.cc:2454] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6926622074dc441dba9d187fd3bfa8f2 in term 1.
I20260927 10:44:54.621387 12587 leader_election.cc:302] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:54.621630 12894 raft_consensus.cc:2781] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:54.621750 12894 raft_consensus.cc:686] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Running, Role: LEADER
I20260927 10:44:54.621893 12894 consensus_queue.cc:237] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:54.623687 12356 catalog_manager.cc:5495] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194). New cstate: current_term: 1 leader_uuid: "6926622074dc441dba9d187fd3bfa8f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } health_report { overall_health: UNKNOWN } } }
I20260927 10:44:54.643983 12792 raft_consensus.cc:1261] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Refusing update from remote peer 6926622074dc441dba9d187fd3bfa8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:54.644250 12513 raft_consensus.cc:1261] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Refusing update from remote peer 6926622074dc441dba9d187fd3bfa8f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:54.644344 12894 consensus_queue.cc:1035] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:54.644704 12870 consensus_queue.cc:1035] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:44:54.658600 12889 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:44:56.207185 12356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:56.207350 12356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:56.209487 12356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Sending DeleteTablet for 3 replicas of tablet 74ff2fd698694215a622445132724ebe
I20260927 10:44:56.210034 12773 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ff2fd698694215a622445132724ebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:56 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:52238
I20260927 10:44:56.210079 12493 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ff2fd698694215a622445132724ebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:56 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:58352
I20260927 10:44:56.210068 12633 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ff2fd698694215a622445132724ebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:56 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:40536
I20260927 10:44:56.210394 12936 tablet_replica.cc:330] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: stopping tablet replica
I20260927 10:44:56.210525 12936 raft_consensus.cc:2229] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:56.210708 12936 raft_consensus.cc:2258] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:56.210943 12934 tablet_replica.cc:330] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: stopping tablet replica
I20260927 10:44:56.211092 12934 raft_consensus.cc:2229] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:56.211230 12934 raft_consensus.cc:2258] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:56.211346 12936 ts_tablet_manager.cc:1905] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:56.211468 12935 tablet_replica.cc:330] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: stopping tablet replica
I20260927 10:44:56.211628 12935 raft_consensus.cc:2229] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:56.211735 12935 raft_consensus.cc:2258] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:56.212273 12934 ts_tablet_manager.cc:1905] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:56.213881 12935 ts_tablet_manager.cc:1905] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:56.213930 12936 ts_tablet_manager.cc:1918] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:44:56.214000 12936 log.cc:1196] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/74ff2fd698694215a622445132724ebe
I20260927 10:44:56.214311 12936 ts_tablet_manager.cc:1939] T 74ff2fd698694215a622445132724ebe P 6926622074dc441dba9d187fd3bfa8f2: Deleting consensus metadata
I20260927 10:44:56.214996 12343 catalog_manager.cc:4841] TS 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539): tablet 74ff2fd698694215a622445132724ebe (table test-table [id=c494ce9efd9d4419b234df1356c89bfc]) successfully deleted
I20260927 10:44:56.215130 12934 ts_tablet_manager.cc:1918] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:44:56.215197 12934 log.cc:1196] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/74ff2fd698694215a622445132724ebe
I20260927 10:44:56.215462 12934 ts_tablet_manager.cc:1939] T 74ff2fd698694215a622445132724ebe P 5611c30d0506451286bde974954dfd81: Deleting consensus metadata
I20260927 10:44:56.216041 12343 catalog_manager.cc:4841] TS 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287): tablet 74ff2fd698694215a622445132724ebe (table test-table [id=c494ce9efd9d4419b234df1356c89bfc]) successfully deleted
I20260927 10:44:56.216521 12935 ts_tablet_manager.cc:1918] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:44:56.216606 12935 log.cc:1196] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/74ff2fd698694215a622445132724ebe
I20260927 10:44:56.216833 12935 ts_tablet_manager.cc:1939] T 74ff2fd698694215a622445132724ebe P 6ac98db4b6514695902e473e5551d96c: Deleting consensus metadata
I20260927 10:44:56.217639 12340 catalog_manager.cc:4841] TS 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571): tablet 74ff2fd698694215a622445132724ebe (table test-table [id=c494ce9efd9d4419b234df1356c89bfc]) successfully deleted
Sep 27 10:44:56 dist-test-slave-wmg0 krb5kdc[12311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505896, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:44:56 dist-test-slave-wmg0 krb5kdc[12311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505896, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:56.246245 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.230504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.10.96.254:32899)
0927 10:44:56.230836 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.230845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.230863 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:44:56.230923 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.230931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.230974 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.231052 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:56.231789 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.232328 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.233157 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.233382 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.233540 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.243060 (+ 9520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:56.243064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:56.243066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:56.243091 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:56.244613 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:56.245023 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:56.245025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:56.245027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:56.245101 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:56.245377 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:56.245379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:56.245380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:56.245617 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:56.245735 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.245931 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.246031 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
I20260927 10:44:56.249145 12356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:44:56.249752 12356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:56.254316 12493 tablet_service.cc:1467] Processing CreateTablet for tablet 87e580f4b3e54972957d17dedc4cb1dc (DEFAULT_TABLE table=test-table [id=5fe70194617348b2923837e51d4a56d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:56.254365 12633 tablet_service.cc:1467] Processing CreateTablet for tablet 87e580f4b3e54972957d17dedc4cb1dc (DEFAULT_TABLE table=test-table [id=5fe70194617348b2923837e51d4a56d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:56.254415 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 87e580f4b3e54972957d17dedc4cb1dc (DEFAULT_TABLE table=test-table [id=5fe70194617348b2923837e51d4a56d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:56.254647 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e580f4b3e54972957d17dedc4cb1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:56.254647 12493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e580f4b3e54972957d17dedc4cb1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:56.254647 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e580f4b3e54972957d17dedc4cb1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:56.256696 12945 tablet_bootstrap.cc:492] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Bootstrap starting.
I20260927 10:44:56.257370 12947 tablet_bootstrap.cc:492] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Bootstrap starting.
I20260927 10:44:56.257617 12945 tablet_bootstrap.cc:654] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:56.258325 12947 tablet_bootstrap.cc:654] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:56.258777 12945 tablet_bootstrap.cc:492] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: No bootstrap required, opened a new log
I20260927 10:44:56.258889 12945 ts_tablet_manager.cc:1397] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:44:56.259019 12946 tablet_bootstrap.cc:492] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Bootstrap starting.
I20260927 10:44:56.259306 12945 raft_consensus.cc:348] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.259425 12945 raft_consensus.cc:374] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:56.259459 12945 raft_consensus.cc:729] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Initialized, Role: FOLLOWER
I20260927 10:44:56.259536 12947 tablet_bootstrap.cc:492] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: No bootstrap required, opened a new log
I20260927 10:44:56.259583 12945 consensus_queue.cc:260] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.259792 12947 ts_tablet_manager.cc:1397] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:56.259872 12946 tablet_bootstrap.cc:654] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:56.260298 12945 ts_tablet_manager.cc:1428] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:44:56.260267 12947 raft_consensus.cc:348] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.260381 12947 raft_consensus.cc:374] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:56.260428 12947 raft_consensus.cc:729] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac98db4b6514695902e473e5551d96c, State: Initialized, Role: FOLLOWER
I20260927 10:44:56.260670 12947 consensus_queue.cc:260] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.260797 12951 raft_consensus.cc:482] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:56.260882 12951 raft_consensus.cc:504] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.261022 12946 tablet_bootstrap.cc:492] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: No bootstrap required, opened a new log
I20260927 10:44:56.261086 12946 ts_tablet_manager.cc:1397] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:56.261502 12946 raft_consensus.cc:348] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.261615 12946 raft_consensus.cc:374] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:56.261646 12946 raft_consensus.cc:729] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5611c30d0506451286bde974954dfd81, State: Initialized, Role: FOLLOWER
I20260927 10:44:56.261653 12947 ts_tablet_manager.cc:1428] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:44:56.261756 12946 consensus_queue.cc:260] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.262092 12946 ts_tablet_manager.cc:1428] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:56.262140 12951 leader_election.cc:288] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539)
I20260927 10:44:56.267269 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.263249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.10.96.195:56049)
0927 10:44:56.263555 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.263569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.263713 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.263933 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.263936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.263951 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:56.264061 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.264068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.264880 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.264883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.266030 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.266039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.266835 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.266839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.266936 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.266959 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.267055 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.267142 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260927 10:44:56.267664 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.263329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56049 (local address 127.10.96.194:34539)
0927 10:44:56.263571 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.263581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.263682 (+ 101us) server_negotiation.cc:407] Connection header received
0927 10:44:56.263790 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.263793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.263830 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.263916 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:56.264193 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.264763 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.266141 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.266690 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.266839 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.266890 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.267474 (+ 584us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.267559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260927 10:44:56.268036 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e580f4b3e54972957d17dedc4cb1dc" candidate_uuid: "6ac98db4b6514695902e473e5551d96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6926622074dc441dba9d187fd3bfa8f2" is_pre_election: true
I20260927 10:44:56.268214 12653 raft_consensus.cc:2454] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac98db4b6514695902e473e5551d96c in term 0.
I20260927 10:44:56.268554 12727 leader_election.cc:302] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6926622074dc441dba9d187fd3bfa8f2, 6ac98db4b6514695902e473e5551d96c; no voters:
I20260927 10:44:56.268783 12951 raft_consensus.cc:2781] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:56.268831 12951 raft_consensus.cc:482] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:56.268868 12951 raft_consensus.cc:3037] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:56.269871 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.263554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:59285 (local address 127.10.96.193:43287)
0927 10:44:56.264847 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.264856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.264882 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:44:56.264930 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.264932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.264978 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.265055 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:56.265326 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.266114 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.267440 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.268540 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.268656 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.268701 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.269671 (+ 970us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.269746 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":94,"threads_started":1}
I20260927 10:44:56.269914 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.262894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43287 (local address 127.10.96.195:59285)
0927 10:44:56.263120 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.263132 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.263241 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.265095 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.265098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.265116 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:56.265233 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.265239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.266271 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.266274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.267280 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.267290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.268658 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.268661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.269547 (+ 886us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.269573 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.269663 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.269753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260927 10:44:56.269856 12951 raft_consensus.cc:504] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.270190 12951 leader_election.cc:288] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [CANDIDATE]: Term 1 election: Requested vote from peers 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287), 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539)
I20260927 10:44:56.270454 12512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e580f4b3e54972957d17dedc4cb1dc" candidate_uuid: "6ac98db4b6514695902e473e5551d96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81" is_pre_election: true
I20260927 10:44:56.270459 12513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e580f4b3e54972957d17dedc4cb1dc" candidate_uuid: "6ac98db4b6514695902e473e5551d96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5611c30d0506451286bde974954dfd81"
I20260927 10:44:56.270516 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e580f4b3e54972957d17dedc4cb1dc" candidate_uuid: "6ac98db4b6514695902e473e5551d96c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6926622074dc441dba9d187fd3bfa8f2"
I20260927 10:44:56.270571 12512 raft_consensus.cc:2454] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ac98db4b6514695902e473e5551d96c in term 0.
I20260927 10:44:56.270598 12653 raft_consensus.cc:3037] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:56.271700 12653 raft_consensus.cc:2454] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ac98db4b6514695902e473e5551d96c in term 1.
I20260927 10:44:56.271966 12727 leader_election.cc:302] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6926622074dc441dba9d187fd3bfa8f2, 6ac98db4b6514695902e473e5551d96c; no voters: 5611c30d0506451286bde974954dfd81
I20260927 10:44:56.272198 12951 raft_consensus.cc:2781] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:56.272292 12951 raft_consensus.cc:686] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 1 LEADER]: Becoming Leader. State: Replica: 6ac98db4b6514695902e473e5551d96c, State: Running, Role: LEADER
I20260927 10:44:56.272500 12951 consensus_queue.cc:237] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.274655 12355 catalog_manager.cc:5495] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ac98db4b6514695902e473e5551d96c (127.10.96.195). New cstate: current_term: 1 leader_uuid: "6ac98db4b6514695902e473e5551d96c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } health_report { overall_health: HEALTHY } } }
I20260927 10:44:56.293963 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.290511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.10.96.195:35571)
0927 10:44:56.290785 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.290792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.290803 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:44:56.290874 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.290877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.290926 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.291028 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:56.291282 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.291811 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.292630 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.292846 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.292922 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.293131 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.293609 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.293696 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.293783 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.293826 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":135,"threads_started":1}
I20260927 10:44:56.303838 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.300773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:35105 (local address 127.10.96.194:34539)
0927 10:44:56.300893 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.300897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.301163 (+ 266us) server_negotiation.cc:407] Connection header received
0927 10:44:56.301215 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.301217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.301259 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.301337 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:56.301598 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.302101 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.302922 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.303123 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.303199 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.303456 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.303518 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.303569 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.303682 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.303726 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:44:56.303869 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.300666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.10.96.195:35105)
0927 10:44:56.300936 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.300948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.301036 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.301357 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.301359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.301364 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:56.301494 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.301502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.302195 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.302197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.302812 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.302821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.303225 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.303229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.303307 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.303349 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.303585 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.303589 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.303677 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.303749 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":104,"threads_started":1}
I20260927 10:44:56.309659 12513 raft_consensus.cc:3037] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:56.309896 12653 raft_consensus.cc:1261] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Refusing update from remote peer 6ac98db4b6514695902e473e5551d96c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:56.310374 12951 consensus_queue.cc:1035] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:56.310918 12513 raft_consensus.cc:1261] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Refusing update from remote peer 6ac98db4b6514695902e473e5551d96c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:56.311268 12951 consensus_queue.cc:1035] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:56.327863 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.324643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:42917 (local address 127.10.96.195:35571)
0927 10:44:56.324751 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.324755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.324926 (+ 171us) server_negotiation.cc:407] Connection header received
0927 10:44:56.325020 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.325022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.325060 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.325119 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:56.325266 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.325848 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.326806 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.327084 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.327198 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.327461 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.327546 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.327622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.327725 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.327767 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:44:56.327880 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.324607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35571 (local address 127.10.96.194:42917)
0927 10:44:56.324856 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.324874 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.324954 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.325121 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.325123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.325128 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:56.325207 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.325212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.325954 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.325957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.326691 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.326701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.327200 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.327207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.327307 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.327342 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.327626 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.327632 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.327691 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.327772 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":109,"threads_started":1}
I20260927 10:44:56.349109 12355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:56.349320 12355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:56.351470 12355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Sending DeleteTablet for 3 replicas of tablet 87e580f4b3e54972957d17dedc4cb1dc
I20260927 10:44:56.352074 12773 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e580f4b3e54972957d17dedc4cb1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:56 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:52238
I20260927 10:44:56.352068 12493 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e580f4b3e54972957d17dedc4cb1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:56 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:58352
I20260927 10:44:56.352254 12934 tablet_replica.cc:330] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: stopping tablet replica
I20260927 10:44:56.352255 12935 tablet_replica.cc:330] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: stopping tablet replica
I20260927 10:44:56.352370 12934 raft_consensus.cc:2229] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:56.352381 12935 raft_consensus.cc:2229] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:56.352481 12934 raft_consensus.cc:2258] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:56.352648 12935 raft_consensus.cc:2258] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:56.352697 12355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:44:56.352972 12633 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e580f4b3e54972957d17dedc4cb1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:56 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:40536
W20260927 10:44:56.353085 12355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:44:56.353152 12936 tablet_replica.cc:330] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: stopping tablet replica
I20260927 10:44:56.353286 12936 raft_consensus.cc:2229] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:56.353392 12936 raft_consensus.cc:2258] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:56.353609 12934 ts_tablet_manager.cc:1905] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:56.354101 12935 ts_tablet_manager.cc:1905] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:56.355157 12936 ts_tablet_manager.cc:1905] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:56.356168 12934 ts_tablet_manager.cc:1918] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:56.356233 12934 log.cc:1196] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/87e580f4b3e54972957d17dedc4cb1dc
I20260927 10:44:56.356503 12934 ts_tablet_manager.cc:1939] T 87e580f4b3e54972957d17dedc4cb1dc P 5611c30d0506451286bde974954dfd81: Deleting consensus metadata
I20260927 10:44:56.356508 12935 ts_tablet_manager.cc:1918] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:56.356570 12935 log.cc:1196] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/87e580f4b3e54972957d17dedc4cb1dc
I20260927 10:44:56.356822 12935 ts_tablet_manager.cc:1939] T 87e580f4b3e54972957d17dedc4cb1dc P 6ac98db4b6514695902e473e5551d96c: Deleting consensus metadata
I20260927 10:44:56.357182 12343 catalog_manager.cc:4841] TS 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287): tablet 87e580f4b3e54972957d17dedc4cb1dc (table test-table [id=5fe70194617348b2923837e51d4a56d7]) successfully deleted
I20260927 10:44:56.357417 12340 catalog_manager.cc:4841] TS 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571): tablet 87e580f4b3e54972957d17dedc4cb1dc (table test-table [id=5fe70194617348b2923837e51d4a56d7]) successfully deleted
I20260927 10:44:56.357686 12936 ts_tablet_manager.cc:1918] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:44:56.357759 12936 log.cc:1196] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/87e580f4b3e54972957d17dedc4cb1dc
I20260927 10:44:56.357998 12936 ts_tablet_manager.cc:1939] T 87e580f4b3e54972957d17dedc4cb1dc P 6926622074dc441dba9d187fd3bfa8f2: Deleting consensus metadata
I20260927 10:44:56.359020 12343 catalog_manager.cc:4841] TS 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539): tablet 87e580f4b3e54972957d17dedc4cb1dc (table test-table [id=5fe70194617348b2923837e51d4a56d7]) successfully deleted
I20260927 10:44:56.360663 12493 tablet_service.cc:1467] Processing CreateTablet for tablet 7a18f6326cde4e93b7a0678f7993bdc6 (DEFAULT_TABLE table=test-table [id=8b2dc417a96e4dfabb1c983947918422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:56.360771 12633 tablet_service.cc:1467] Processing CreateTablet for tablet 7a18f6326cde4e93b7a0678f7993bdc6 (DEFAULT_TABLE table=test-table [id=8b2dc417a96e4dfabb1c983947918422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:56.360905 12493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a18f6326cde4e93b7a0678f7993bdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:56.361047 12633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a18f6326cde4e93b7a0678f7993bdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:56.362921 12945 tablet_bootstrap.cc:492] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Bootstrap starting.
I20260927 10:44:56.363062 12946 tablet_bootstrap.cc:492] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81: Bootstrap starting.
I20260927 10:44:56.363097 12773 tablet_service.cc:1467] Processing CreateTablet for tablet 7a18f6326cde4e93b7a0678f7993bdc6 (DEFAULT_TABLE table=test-table [id=8b2dc417a96e4dfabb1c983947918422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:44:56.363324 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a18f6326cde4e93b7a0678f7993bdc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:56.363721 12945 tablet_bootstrap.cc:654] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:56.363907 12946 tablet_bootstrap.cc:654] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:56.365307 12946 tablet_bootstrap.cc:492] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81: No bootstrap required, opened a new log
I20260927 10:44:56.365387 12946 ts_tablet_manager.cc:1397] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:56.365818 12946 raft_consensus.cc:348] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.365929 12946 raft_consensus.cc:374] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:56.365967 12946 raft_consensus.cc:729] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5611c30d0506451286bde974954dfd81, State: Initialized, Role: FOLLOWER
I20260927 10:44:56.366071 12946 consensus_queue.cc:260] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.366266 12945 tablet_bootstrap.cc:492] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: No bootstrap required, opened a new log
I20260927 10:44:56.366307 12946 ts_tablet_manager.cc:1428] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:56.366370 12945 ts_tablet_manager.cc:1397] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 10:44:56.366513 12947 tablet_bootstrap.cc:492] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Bootstrap starting.
I20260927 10:44:56.366727 12945 raft_consensus.cc:348] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.366859 12945 raft_consensus.cc:374] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:56.366895 12945 raft_consensus.cc:729] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6926622074dc441dba9d187fd3bfa8f2, State: Initialized, Role: FOLLOWER
I20260927 10:44:56.367013 12945 consensus_queue.cc:260] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.367262 12945 ts_tablet_manager.cc:1428] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:44:56.367374 12947 tablet_bootstrap.cc:654] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:56.368805 12947 tablet_bootstrap.cc:492] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: No bootstrap required, opened a new log
I20260927 10:44:56.368881 12947 ts_tablet_manager.cc:1397] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 10:44:56.369264 12947 raft_consensus.cc:348] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.369365 12947 raft_consensus.cc:374] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:56.369401 12947 raft_consensus.cc:729] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac98db4b6514695902e473e5551d96c, State: Initialized, Role: FOLLOWER
I20260927 10:44:56.369529 12947 consensus_queue.cc:260] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.369789 12947 ts_tablet_manager.cc:1428] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:44:56.421578 12953 raft_consensus.cc:482] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:44:56.421727 12953 raft_consensus.cc:504] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.422850 12953 leader_election.cc:288] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:56.427461 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.423389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:33881 (local address 127.10.96.195:35571)
0927 10:44:56.423504 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.423509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.423695 (+ 186us) server_negotiation.cc:407] Connection header received
0927 10:44:56.423855 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.423859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.423905 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.423943 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:56.424314 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.425058 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.426395 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.426896 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.427020 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.427073 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.427264 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.427334 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:44:56.427508 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.423102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35571 (local address 127.10.96.193:33881)
0927 10:44:56.423595 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.423607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.423714 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.424077 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.424080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.424096 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:56.424198 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.424205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.425192 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.425196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.426291 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.426298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.427007 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.427011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.427124 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.427152 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.427249 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.427323 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":113,"threads_started":1}
I20260927 10:44:56.427538 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.423019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.10.96.193:44273)
0927 10:44:56.423273 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.423290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.423401 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.423776 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.423779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.423801 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:44:56.423928 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.423934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.424780 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.424787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.426036 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.426046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.427069 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.427073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.427176 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.427197 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.427266 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.427349 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260927 10:44:56.427882 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.423380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:44273 (local address 127.10.96.194:34539)
0927 10:44:56.423517 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.423521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.423535 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:44:56.423587 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.423590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.423662 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.423734 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:44:56.424052 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.424669 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.426151 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.426794 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.426927 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.426979 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.427693 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.427768 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:44:56.428284 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18f6326cde4e93b7a0678f7993bdc6" candidate_uuid: "5611c30d0506451286bde974954dfd81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6926622074dc441dba9d187fd3bfa8f2" is_pre_election: true
I20260927 10:44:56.428332 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18f6326cde4e93b7a0678f7993bdc6" candidate_uuid: "5611c30d0506451286bde974954dfd81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c" is_pre_election: true
I20260927 10:44:56.428419 12653 raft_consensus.cc:2454] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5611c30d0506451286bde974954dfd81 in term 0.
I20260927 10:44:56.428505 12792 raft_consensus.cc:2454] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5611c30d0506451286bde974954dfd81 in term 0.
I20260927 10:44:56.428803 12447 leader_election.cc:302] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:56.429291 12953 raft_consensus.cc:2781] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:44:56.429358 12953 raft_consensus.cc:482] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:44:56.429412 12953 raft_consensus.cc:3037] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:56.430609 12953 raft_consensus.cc:504] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.430946 12953 leader_election.cc:288] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [CANDIDATE]: Term 1 election: Requested vote from peers 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539), 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571)
I20260927 10:44:56.431242 12653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18f6326cde4e93b7a0678f7993bdc6" candidate_uuid: "5611c30d0506451286bde974954dfd81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6926622074dc441dba9d187fd3bfa8f2"
I20260927 10:44:56.431295 12792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a18f6326cde4e93b7a0678f7993bdc6" candidate_uuid: "5611c30d0506451286bde974954dfd81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac98db4b6514695902e473e5551d96c"
I20260927 10:44:56.431367 12653 raft_consensus.cc:3037] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:56.431394 12792 raft_consensus.cc:3037] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:56.432447 12653 raft_consensus.cc:2454] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5611c30d0506451286bde974954dfd81 in term 1.
I20260927 10:44:56.432447 12792 raft_consensus.cc:2454] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5611c30d0506451286bde974954dfd81 in term 1.
I20260927 10:44:56.432713 12447 leader_election.cc:302] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5611c30d0506451286bde974954dfd81, 6926622074dc441dba9d187fd3bfa8f2; no voters:
I20260927 10:44:56.432876 12953 raft_consensus.cc:2781] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:56.433135 12953 raft_consensus.cc:686] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 1 LEADER]: Becoming Leader. State: Replica: 5611c30d0506451286bde974954dfd81, State: Running, Role: LEADER
I20260927 10:44:56.433390 12953 consensus_queue.cc:237] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } }
I20260927 10:44:56.435633 12356 catalog_manager.cc:5495] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5611c30d0506451286bde974954dfd81 (127.10.96.193). New cstate: current_term: 1 leader_uuid: "5611c30d0506451286bde974954dfd81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5611c30d0506451286bde974954dfd81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 } health_report { overall_health: UNKNOWN } } }
I20260927 10:44:56.457547 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.453138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.10.96.193:43287)
0927 10:44:56.453283 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.453287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.453300 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:44:56.453420 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.453423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.453507 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.453600 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:56.453869 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.454518 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.455738 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.455964 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.456043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.456308 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.456738 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.456830 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.457266 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.457359 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:44:56.468552 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.464722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:46623 (local address 127.10.96.194:34539)
0927 10:44:56.464842 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.464846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.464964 (+ 118us) server_negotiation.cc:407] Connection header received
0927 10:44:56.465086 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.465088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.465128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.465192 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:56.465775 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.466258 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.467452 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.467664 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.467763 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.468086 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.468165 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.468246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.468390 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.468438 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:44:56.468608 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.464542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34539 (local address 127.10.96.193:46623)
0927 10:44:56.464884 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.464899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.465000 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.465216 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.465219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.465226 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:56.465672 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.465681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.466374 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.466377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.467331 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.467341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.467789 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.467793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.467888 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.467938 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.468277 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.468284 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.468361 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.468444 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":87,"threads_started":1}
I20260927 10:44:56.475283 12792 raft_consensus.cc:1261] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Refusing update from remote peer 5611c30d0506451286bde974954dfd81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:56.475304 12653 raft_consensus.cc:1261] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Refusing update from remote peer 5611c30d0506451286bde974954dfd81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:44:56.475817 12953 consensus_queue.cc:1035] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac98db4b6514695902e473e5551d96c" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:56.476195 12953 consensus_queue.cc:1035] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6926622074dc441dba9d187fd3bfa8f2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:44:56.500139 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.495998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43287 (local address 127.10.96.194:39733)
0927 10:44:56.496113 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:56.496125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:56.496222 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:56.496563 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:56.496566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:56.496575 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:44:56.496673 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.496680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.497754 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.497758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.498720 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:56.498730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.499404 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.499408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:56.499484 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.499515 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.499838 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.499844 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:44:56.499905 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 10:44:56.500008 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 10:44:56.500488 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:56.496165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:39733 (local address 127.10.96.193:43287)
0927 10:44:56.496300 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:56.496304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:56.496315 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:44:56.496351 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:56.496353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:56.496394 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:56.496457 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:44:56.496800 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.497611 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.498908 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:56.499237 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:56.499332 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:56.499640 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:44:56.499732 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:44:56.499799 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:56.500323 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 10:44:56.500373 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:44:56.501281 12918 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 10:44:58.063616 12355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:58.063760 12355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:44:58.066192 12355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf51cf0edf540c89bec0fc8c43937ea: Sending DeleteTablet for 3 replicas of tablet 7a18f6326cde4e93b7a0678f7993bdc6
I20260927 10:44:58.066726 12493 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18f6326cde4e93b7a0678f7993bdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:58 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:58352
I20260927 10:44:58.066787 12773 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18f6326cde4e93b7a0678f7993bdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:58 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:52238
I20260927 10:44:58.067065 13006 tablet_replica.cc:330] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81: stopping tablet replica
I20260927 10:44:58.067209 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 12428
I20260927 10:44:58.067233 13006 raft_consensus.cc:2229] T 7a18f6326cde4e93b7a0678f7993bdc6 P 5611c30d0506451286bde974954dfd81 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:44:58.069060 13007 tablet_replica.cc:330] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: stopping tablet replica
I20260927 10:44:58.069204 13007 raft_consensus.cc:2229] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:58.069334 13007 raft_consensus.cc:2258] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:58.069350 12633 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a18f6326cde4e93b7a0678f7993bdc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:44:58 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:40536
I20260927 10:44:58.069839 13008 tablet_replica.cc:330] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: stopping tablet replica
I20260927 10:44:58.069965 13008 raft_consensus.cc:2229] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:44:58.070035 13007 ts_tablet_manager.cc:1905] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:58.070081 13008 raft_consensus.cc:2258] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:44:58.070622 13008 ts_tablet_manager.cc:1905] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:44:58.072800 13007 ts_tablet_manager.cc:1918] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:44:58.072882 13007 log.cc:1196] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/7a18f6326cde4e93b7a0678f7993bdc6
I20260927 10:44:58.073211 13007 ts_tablet_manager.cc:1939] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6ac98db4b6514695902e473e5551d96c: Deleting consensus metadata
I20260927 10:44:58.073258 13008 ts_tablet_manager.cc:1918] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 10:44:58.073319 13008 log.cc:1196] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/7a18f6326cde4e93b7a0678f7993bdc6
I20260927 10:44:58.073642 13008 ts_tablet_manager.cc:1939] T 7a18f6326cde4e93b7a0678f7993bdc6 P 6926622074dc441dba9d187fd3bfa8f2: Deleting consensus metadata
I20260927 10:44:58.073993 12340 catalog_manager.cc:4841] TS 6ac98db4b6514695902e473e5551d96c (127.10.96.195:35571): tablet 7a18f6326cde4e93b7a0678f7993bdc6 (table test-table [id=8b2dc417a96e4dfabb1c983947918422]) successfully deleted
I20260927 10:44:58.074287 12343 catalog_manager.cc:4841] TS 6926622074dc441dba9d187fd3bfa8f2 (127.10.96.194:34539): tablet 7a18f6326cde4e93b7a0678f7993bdc6 (table test-table [id=8b2dc417a96e4dfabb1c983947918422]) successfully deleted
W20260927 10:44:58.075822 12343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:43287 (error 108)
I20260927 10:44:58.076066 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 12568
I20260927 10:44:58.077358 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:58.076939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43287 (local address 127.0.0.1:58372)
0927 10:44:58.077181 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:58.077268 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
W20260927 10:44:58.077534 12343 catalog_manager.cc:4584] TS 5611c30d0506451286bde974954dfd81 (127.10.96.193:43287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a18f6326cde4e93b7a0678f7993bdc6: Network error: Client connection negotiation failed: client connection to 127.10.96.193:43287: connect: Connection refused (error 111)
I20260927 10:44:58.084434 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 12708
I20260927 10:44:58.090692 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 12324
2026-09-27T10:44:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6442 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:44:58 dist-test-slave-wmg0 krb5kdc[13013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:44:58 dist-test-slave-wmg0 krb5kdc[13013](info): set up 2 sockets
Sep 27 10:44:58 dist-test-slave-wmg0 krb5kdc[13013](info): commencing operation
krb5kdc: starting...
W20260927 10:44:59.125921 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:44:59.241962 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:40947
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:59.352684 13026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:59.352969 13026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:59.353050 13026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:59.357018 13026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:44:59.357108 13026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:59.357164 13026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:59.357226 13026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:44:59.357262 13026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:44:59.362512 13026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:40947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:59.363545 13026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:59.364547 13026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:59.372758 13032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:59.372848 13026 server_base.cc:1034] running on GCE node
W20260927 10:44:59.372769 13031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:59.372769 13034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:59.373605 13026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:59.374653 13026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:59.375811 13026 hybrid_clock.cc:648] HybridClock initialized: now 1790505899375796 us; error 31 us; skew 500 ppm
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:59.378607 13026 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:44:59.379487 13026 webserver.cc:460] Webserver started at http://127.10.96.254:33125/ using document root <none> and password file <none>
I20260927 10:44:59.380023 13026 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:59.380079 13026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:59.380316 13026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:59.382426 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "53f836d0839c47eca48d7fe50c46e21a"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "03dab9c7c9cfa4f74aa2eab2ae008977"
server_key_iv: "fa57e8cdad886aa3b6994bbcf39072dc"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.383013 13026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "53f836d0839c47eca48d7fe50c46e21a"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "03dab9c7c9cfa4f74aa2eab2ae008977"
server_key_iv: "fa57e8cdad886aa3b6994bbcf39072dc"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.386513 13026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:44:59.388720 13041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.389897 13026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:44:59.390013 13026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "53f836d0839c47eca48d7fe50c46e21a"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "03dab9c7c9cfa4f74aa2eab2ae008977"
server_key_iv: "fa57e8cdad886aa3b6994bbcf39072dc"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.390110 13026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:59.435523 13026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:59.439076 13026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:59.439289 13026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:59.446310 13026 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:40947
I20260927 10:44:59.446338 13093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:40947 every 8 connection(s)
I20260927 10:44:59.447243 13026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:44:59.448213 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13026
I20260927 10:44:59.448325 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:44:59.448611 10627 external_mini_cluster.cc:1428] Setting key 29f093ede3e58edd6088c098842aa35d
I20260927 10:44:59.450569 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:44:59.455924 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a: Bootstrap starting.
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:59.458441 13094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a: Neither blocks nor log segments found. Creating new log.
I20260927 10:44:59.459247 13094 log.cc:824] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:44:59.461336 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a: No bootstrap required, opened a new log
I20260927 10:44:59.462325 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:59.450032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.10.96.254:40947)
0927 10:44:59.450444 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:59.450458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:59.450491 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:44:59.451142 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:59.451159 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:59.451477 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:59.451815 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:59.452562 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.453591 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.454421 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.454882 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.455053 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:59.457828 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:59.457832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:59.457845 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:59.457869 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:59.459661 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:59.460139 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:59.460144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:59.460147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:59.460248 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:59.460525 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:59.460527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:59.460528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:59.460882 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:59.461015 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:59.461313 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 10:44:59.461646 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260927 10:44:59.464461 13094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f836d0839c47eca48d7fe50c46e21a" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40947 } }
I20260927 10:44:59.464718 13094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:44:59.464787 13094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f836d0839c47eca48d7fe50c46e21a, State: Initialized, Role: FOLLOWER
I20260927 10:44:59.465229 13094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f836d0839c47eca48d7fe50c46e21a" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40947 } }
I20260927 10:44:59.465372 13094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:44:59.465456 13094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:44:59.465577 13094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:44:59.466390 13094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f836d0839c47eca48d7fe50c46e21a" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40947 } }
I20260927 10:44:59.466711 13094 leader_election.cc:302] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53f836d0839c47eca48d7fe50c46e21a; no voters:
I20260927 10:44:59.466989 13094 leader_election.cc:288] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:44:59.467119 13099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:44:59.467329 13099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [term 1 LEADER]: Becoming Leader. State: Replica: 53f836d0839c47eca48d7fe50c46e21a, State: Running, Role: LEADER
I20260927 10:44:59.467653 13099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f836d0839c47eca48d7fe50c46e21a" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40947 } }
I20260927 10:44:59.468116 13094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:44:59.469208 13101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53f836d0839c47eca48d7fe50c46e21a. Latest consensus state: current_term: 1 leader_uuid: "53f836d0839c47eca48d7fe50c46e21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f836d0839c47eca48d7fe50c46e21a" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40947 } } }
I20260927 10:44:59.469331 13101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:59.469569 13100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53f836d0839c47eca48d7fe50c46e21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53f836d0839c47eca48d7fe50c46e21a" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40947 } } }
I20260927 10:44:59.469655 13100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a [sys.catalog]: This master's current role is: LEADER
I20260927 10:44:59.469758 13108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:44:59.473141 13108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:44:59.478395 13108 catalog_manager.cc:1347] Generated new cluster ID: dbd8ea223ef446fd8bd6f21813881526
I20260927 10:44:59.478461 13108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:44:59.500399 13108 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:44:59.501333 13108 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:44:59.505692 13108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53f836d0839c47eca48d7fe50c46e21a: Generated new TSK 0
I20260927 10:44:59.506335 13108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:44:59.533890 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40947
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:59.643234 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:59.643496 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:59.643537 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:59.647490 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:59.647543 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:59.647636 13120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:44:59.652760 13120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:59.653793 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:59.654801 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:59.664191 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:59.664182 13125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:59.664168 13126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:59.664523 13120 server_base.cc:1034] running on GCE node
I20260927 10:44:59.664952 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:59.665634 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:59.666814 13120 hybrid_clock.cc:648] HybridClock initialized: now 1790505899666798 us; error 26 us; skew 500 ppm
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:59.669606 13120 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:44:59.670447 13120 webserver.cc:460] Webserver started at http://127.10.96.193:45319/ using document root <none> and password file <none>
I20260927 10:44:59.670984 13120 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:59.671069 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:59.671271 13120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:59.673238 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "c3377d1cfc3446a685b2c4d95a8aed38"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "c6ecc61c69dc64cab5e99eccb4173b93"
server_key_iv: "96fe2f71581e0921a3d02f153d1e0c8b"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.673837 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "c3377d1cfc3446a685b2c4d95a8aed38"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "c6ecc61c69dc64cab5e99eccb4173b93"
server_key_iv: "96fe2f71581e0921a3d02f153d1e0c8b"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.677295 13120 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:59.679632 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.680799 13120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:59.680936 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "c3377d1cfc3446a685b2c4d95a8aed38"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "c6ecc61c69dc64cab5e99eccb4173b93"
server_key_iv: "96fe2f71581e0921a3d02f153d1e0c8b"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.681046 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:59.702612 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:59.705420 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:59.705647 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:59.706238 13120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:59.707229 13120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:59.707304 13120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.707368 13120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:59.707397 13120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.716760 13120 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:32965
I20260927 10:44:59.717585 13248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:32965 every 8 connection(s)
I20260927 10:44:59.717953 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:44:59.719875 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13120
I20260927 10:44:59.719985 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:44:59.720247 10627 external_mini_cluster.cc:1428] Setting key ecc6ec3643f64ee09fc3b4e69e3d11b9
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:44:59.732872 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:59.719810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:57131 (local address 127.10.96.254:40947)
0927 10:44:59.719959 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:59.719964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:59.720894 (+ 930us) server_negotiation.cc:407] Connection header received
0927 10:44:59.722307 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:59.722314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:59.722374 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:59.722470 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:59.723587 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.724464 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.725620 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.726031 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.726158 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:59.728800 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:59.728807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:59.728810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:59.728845 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:59.730418 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:59.730893 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:59.730895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:59.730897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:59.731017 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:59.731428 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:59.731430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:59.731432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:59.731607 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:59.731689 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:59.732442 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 10:44:59.732616 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:44:59.733559 13251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:59.720251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:40947 (local address 127.10.96.193:57131)
0927 10:44:59.720742 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:59.720793 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:59.722099 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:59.722616 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:59.722623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:59.722983 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:59.723441 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:59.723455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.724586 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.724588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:59.725452 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:59.725518 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.726172 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.726175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:59.726314 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:59.726963 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:59.726979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:59.728632 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:59.730519 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:59.730524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:59.730537 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:59.730788 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:59.731150 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:59.731157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:59.731160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:59.731332 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:59.731712 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:59.731717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:59.732071 (+ 354us) client_negotiation.cc:769] Sending connection context
0927 10:44:59.732236 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 10:44:59.732569 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
I20260927 10:44:59.735038 13249 heartbeater.cc:347] Connected to a master server at 127.10.96.254:40947
I20260927 10:44:59.735280 13249 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:59.735940 13249 heartbeater.cc:511] Master 127.10.96.254:40947 requested a full tablet report, sending...
I20260927 10:44:59.737510 13058 ts_manager.cc:194] Registered new tserver with Master: c3377d1cfc3446a685b2c4d95a8aed38 (127.10.96.193:32965)
I20260927 10:44:59.738636 13058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:57131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:44:59.748683 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40947
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:44:59.855700 13254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:44:59.855952 13254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:44:59.855993 13254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:44:59.859995 13254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:44:59.860047 13254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:44:59.860145 13254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:44:59.865206 13254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:44:59.866202 13254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:44:59.867180 13254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:44:59.876551 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:44:59.876611 13262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:59.876781 13254 server_base.cc:1034] running on GCE node
W20260927 10:44:59.876565 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:44:59.877315 13254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:44:59.877964 13254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:44:59.879166 13254 hybrid_clock.cc:648] HybridClock initialized: now 1790505899879121 us; error 42 us; skew 500 ppm
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:44:59.881770 13254 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:44:59.882696 13254 webserver.cc:460] Webserver started at http://127.10.96.194:40947/ using document root <none> and password file <none>
I20260927 10:44:59.883299 13254 fs_manager.cc:359] Metadata directory not provided
I20260927 10:44:59.883390 13254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:44:59.883594 13254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:44:59.885800 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "7aba695c95244dabab81f4fc29b2cec0"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "08e829ecb1269f8d468d9770933991ea"
server_key_iv: "afde3df0e793d96bdf6b602b4e2fd6fa"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.886348 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "7aba695c95244dabab81f4fc29b2cec0"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "08e829ecb1269f8d468d9770933991ea"
server_key_iv: "afde3df0e793d96bdf6b602b4e2fd6fa"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.889835 13254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:44:59.892093 13269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.893348 13254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:44:59.893491 13254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "7aba695c95244dabab81f4fc29b2cec0"
format_stamp: "Formatted at 2026-09-27 10:44:59 on dist-test-slave-wmg0"
server_key: "08e829ecb1269f8d468d9770933991ea"
server_key_iv: "afde3df0e793d96bdf6b602b4e2fd6fa"
server_key_version: "encryptionkey@0"
I20260927 10:44:59.893610 13254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:44:59.923987 13254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:44:59.927028 13254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:44:59.927227 13254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:44:59.927790 13254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:44:59.928586 13254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:44:59.928653 13254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.928721 13254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:44:59.928773 13254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:44:59.937383 13254 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:42781
I20260927 10:44:59.937407 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:42781 every 8 connection(s)
I20260927 10:44:59.938228 13254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:44:59.944923 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13254
I20260927 10:44:59.945030 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:44:59.945353 10627 external_mini_cluster.cc:1428] Setting key 22c203c69b0cb5a76ca7bd5ab913bbc0
Sep 27 10:44:59 dist-test-slave-wmg0 krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505899, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:44:59.952368 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:59.939840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:38907 (local address 127.10.96.254:40947)
0927 10:44:59.939989 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:44:59.939994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:44:59.940780 (+ 786us) server_negotiation.cc:407] Connection header received
0927 10:44:59.942030 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:44:59.942033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:44:59.942077 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:44:59.942151 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:44:59.943199 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.943726 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.944813 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.945156 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.945270 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:59.948271 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:44:59.948273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:44:59.948275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:44:59.948297 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:44:59.950101 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:59.950604 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:59.950610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:59.950615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:59.950715 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:44:59.951109 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:44:59.951116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:44:59.951120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:44:59.951384 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:44:59.951490 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:44:59.951996 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 10:44:59.952137 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:44:59.953039 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:44:59.940230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:40947 (local address 127.10.96.194:38907)
0927 10:44:59.940638 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 10:44:59.940678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:44:59.941848 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:44:59.942301 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:44:59.942307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:44:59.942627 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:44:59.943081 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:59.943094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.943830 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.943833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:59.944699 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:44:59.944708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:44:59.945308 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:44:59.945311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:44:59.945521 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:44:59.946179 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:44:59.946198 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:44:59.948110 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:44:59.950234 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:59.950239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:59.950251 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:59.950501 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:59.950829 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:44:59.950832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:44:59.950834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:44:59.950968 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:44:59.951513 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:44:59.951518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:44:59.951743 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 10:44:59.951904 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 10:44:59.952155 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":112,"threads_started":1}
I20260927 10:44:59.954151 13383 heartbeater.cc:347] Connected to a master server at 127.10.96.254:40947
I20260927 10:44:59.954414 13383 heartbeater.cc:464] Registering TS with master...
I20260927 10:44:59.954867 13383 heartbeater.cc:511] Master 127.10.96.254:40947 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:44:59.955871 13058 ts_manager.cc:194] Registered new tserver with Master: 7aba695c95244dabab81f4fc29b2cec0 (127.10.96.194:42781)
I20260927 10:44:59.956552 13058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:38907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:44:59.972451 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40947
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:00.081563 13388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:00.081844 13388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:00.081914 13388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:00.086140 13388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:00.086283 13388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:00.086436 13388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:00.091791 13388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:00.092815 13388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:00.093917 13388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:00.103108 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:00.103098 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:00.103094 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:00.103359 13388 server_base.cc:1034] running on GCE node
I20260927 10:45:00.103719 13388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:00.104352 13388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:00.105530 13388 hybrid_clock.cc:648] HybridClock initialized: now 1790505900105505 us; error 47 us; skew 500 ppm
Sep 27 10:45:00 dist-test-slave-wmg0 krb5kdc[13013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505900, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:00.108108 13388 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:00.108989 13388 webserver.cc:460] Webserver started at http://127.10.96.195:46109/ using document root <none> and password file <none>
I20260927 10:45:00.109589 13388 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:00.109680 13388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:00.109910 13388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:00.111869 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "b08e71ce3fc141198145932d7ed95cc2"
format_stamp: "Formatted at 2026-09-27 10:45:00 on dist-test-slave-wmg0"
server_key: "398e2bcb198481d549c78340f4c57390"
server_key_iv: "564c4f133ccf38d5c1e2ab74c6e28a56"
server_key_version: "encryptionkey@0"
I20260927 10:45:00.112428 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "b08e71ce3fc141198145932d7ed95cc2"
format_stamp: "Formatted at 2026-09-27 10:45:00 on dist-test-slave-wmg0"
server_key: "398e2bcb198481d549c78340f4c57390"
server_key_iv: "564c4f133ccf38d5c1e2ab74c6e28a56"
server_key_version: "encryptionkey@0"
I20260927 10:45:00.116050 13388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:45:00.118508 13403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:00.119724 13388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:00.119874 13388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "b08e71ce3fc141198145932d7ed95cc2"
format_stamp: "Formatted at 2026-09-27 10:45:00 on dist-test-slave-wmg0"
server_key: "398e2bcb198481d549c78340f4c57390"
server_key_iv: "564c4f133ccf38d5c1e2ab74c6e28a56"
server_key_version: "encryptionkey@0"
I20260927 10:45:00.119987 13388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:00.137799 13388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:00.140648 13388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:00.140857 13388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:00.141444 13388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:00.142287 13388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:00.142355 13388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:00.142431 13388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:00.142474 13388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:00.151309 13388 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:33249
I20260927 10:45:00.151337 13516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:33249 every 8 connection(s)
I20260927 10:45:00.152168 13388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:00.158300 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13388
I20260927 10:45:00.158417 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:00.158697 10627 external_mini_cluster.cc:1428] Setting key 13a401e133aeabff63eda96adeef59ba
Sep 27 10:45:00 dist-test-slave-wmg0 krb5kdc[13013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505900, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:00.165804 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:00.153994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:50607 (local address 127.10.96.254:40947)
0927 10:45:00.154142 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:00.154147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:00.155013 (+ 866us) server_negotiation.cc:407] Connection header received
0927 10:45:00.156207 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:00.156210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:00.156259 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:00.156338 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:00.157592 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:00.158110 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:00.159698 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:00.159924 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:00.160005 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:00.162532 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:00.162538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:00.162545 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:00.162567 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:00.163830 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:00.164322 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:00.164327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:00.164329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:00.164409 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:00.164734 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:00.164739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:00.164741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:00.164926 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:00.165031 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:00.165518 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 10:45:00.165632 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:45:00.166555 13519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:00.154430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:40947 (local address 127.10.96.195:50607)
0927 10:45:00.154855 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:00.154897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:00.156022 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:00.156533 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:00.156540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:00.156886 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:00.157392 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:00.157407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:00.158267 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:00.158271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:00.159595 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:00.159606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:00.160068 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:00.160072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:00.160230 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:00.160884 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:00.160900 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:00.162375 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:00.163944 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:00.163950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:00.163965 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:00.164211 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:00.164499 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:00.164504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:00.164506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:00.164638 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:00.165031 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:00.165038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:00.165277 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 10:45:00.165461 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 10:45:00.165777 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":121,"threads_started":1}
I20260927 10:45:00.167555 13517 heartbeater.cc:347] Connected to a master server at 127.10.96.254:40947
I20260927 10:45:00.167847 13517 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:00.168323 13517 heartbeater.cc:511] Master 127.10.96.254:40947 requested a full tablet report, sending...
I20260927 10:45:00.169325 13058 ts_manager.cc:194] Registered new tserver with Master: b08e71ce3fc141198145932d7ed95cc2 (127.10.96.195:33249)
I20260927 10:45:00.169893 13058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:50607
I20260927 10:45:00.172322 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:45:00.184341 13097 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:00.182986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.10.96.254:40947)
0927 10:45:00.183196 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:00.183199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:00.183294 (+ 95us) server_negotiation.cc:407] Connection header received
0927 10:45:00.183388 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:00.183390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:00.183427 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:00.183497 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:00.184191 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36868: BlockingRecv error: recv got EOF from 127.0.0.1:36868 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 10:45:00.185362 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13120
I20260927 10:45:00.190589 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13254
I20260927 10:45:00.196097 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13388
I20260927 10:45:00.200980 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13026
2026-09-27T10:45:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2108 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:00 dist-test-slave-wmg0 krb5kdc[13529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:00 dist-test-slave-wmg0 krb5kdc[13529](info): set up 2 sockets
Sep 27 10:45:00 dist-test-slave-wmg0 krb5kdc[13529](info): commencing operation
krb5kdc: starting...
W20260927 10:45:01.237763 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:01 dist-test-slave-wmg0 krb5kdc[13529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:01.358615 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37251
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:37251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:01.466941 13542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:01.467216 13542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:01.467257 13542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:01.471374 13542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:01.471428 13542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:01.471447 13542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:01.471463 13542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:01.471525 13542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:01.476712 13542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:37251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:01.477802 13542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:01.478839 13542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:01.487027 13547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:01.487036 13548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:01.487036 13550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:01.487286 13542 server_base.cc:1034] running on GCE node
I20260927 10:45:01.487862 13542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:01.488925 13542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:01.490130 13542 hybrid_clock.cc:648] HybridClock initialized: now 1790505901490117 us; error 28 us; skew 500 ppm
Sep 27 10:45:01 dist-test-slave-wmg0 krb5kdc[13529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505901, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:01.497850 13542 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:01.498746 13542 webserver.cc:460] Webserver started at http://127.10.96.254:46363/ using document root <none> and password file <none>
I20260927 10:45:01.499308 13542 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:01.499366 13542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:01.499527 13542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:01.501595 13542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "219a6370a3c04e5a8a290bbe2e20f258"
format_stamp: "Formatted at 2026-09-27 10:45:01 on dist-test-slave-wmg0"
server_key: "33df5c31b1fdba796659ab0fc10c8898"
server_key_iv: "f7589d2ebd5b8587a10fd4abb7a7e008"
server_key_version: "encryptionkey@0"
I20260927 10:45:01.502117 13542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "219a6370a3c04e5a8a290bbe2e20f258"
format_stamp: "Formatted at 2026-09-27 10:45:01 on dist-test-slave-wmg0"
server_key: "33df5c31b1fdba796659ab0fc10c8898"
server_key_iv: "f7589d2ebd5b8587a10fd4abb7a7e008"
server_key_version: "encryptionkey@0"
I20260927 10:45:01.505847 13542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:45:01.508117 13557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:01.509287 13542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:01.509382 13542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "219a6370a3c04e5a8a290bbe2e20f258"
format_stamp: "Formatted at 2026-09-27 10:45:01 on dist-test-slave-wmg0"
server_key: "33df5c31b1fdba796659ab0fc10c8898"
server_key_iv: "f7589d2ebd5b8587a10fd4abb7a7e008"
server_key_version: "encryptionkey@0"
I20260927 10:45:01.509527 13542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:01.522219 13542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:01.524901 13542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:01.525048 13542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:01.531833 13542 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:37251
I20260927 10:45:01.532116 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:37251 every 8 connection(s)
I20260927 10:45:01.532790 13542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:01.534507 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13542
I20260927 10:45:01.534633 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:01.534917 10627 external_mini_cluster.cc:1428] Setting key 19f5761b9bd790534c738125eb26a2b2
I20260927 10:45:01.535908 13610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:01.542462 13610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258: Bootstrap starting.
Sep 27 10:45:01 dist-test-slave-wmg0 krb5kdc[13529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:01.545182 13610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:01.546164 13610 log.cc:824] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:01.548079 13610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258: No bootstrap required, opened a new log
I20260927 10:45:01.549572 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:01.536257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.10.96.254:37251)
0927 10:45:01.536652 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:01.536670 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:01.536710 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:45:01.537580 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:01.537602 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:01.537998 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:01.538376 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:01.539044 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:01.540486 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:01.541353 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:01.542030 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:01.542189 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:01.544835 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:01.544841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:01.544855 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:01.544886 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:01.546829 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:01.547269 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:01.547281 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:01.547286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:01.547388 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:01.547681 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:01.547684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:01.547686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:01.548060 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:01.548257 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:01.548545 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:45:01.548818 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260927 10:45:01.552137 13610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219a6370a3c04e5a8a290bbe2e20f258" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37251 } }
I20260927 10:45:01.552409 13610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:01.552481 13610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a6370a3c04e5a8a290bbe2e20f258, State: Initialized, Role: FOLLOWER
I20260927 10:45:01.552588 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:45:01.552914 13610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219a6370a3c04e5a8a290bbe2e20f258" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37251 } }
I20260927 10:45:01.553031 13610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:01.553107 13610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:01.553200 13610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:01.554118 13610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219a6370a3c04e5a8a290bbe2e20f258" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37251 } }
I20260927 10:45:01.554446 13610 leader_election.cc:302] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219a6370a3c04e5a8a290bbe2e20f258; no voters:
I20260927 10:45:01.554728 13610 leader_election.cc:288] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:01.554876 13616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:01.555187 13616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [term 1 LEADER]: Becoming Leader. State: Replica: 219a6370a3c04e5a8a290bbe2e20f258, State: Running, Role: LEADER
I20260927 10:45:01.555533 13616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219a6370a3c04e5a8a290bbe2e20f258" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37251 } }
I20260927 10:45:01.555955 13610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:01.556754 13616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219a6370a3c04e5a8a290bbe2e20f258. Latest consensus state: current_term: 1 leader_uuid: "219a6370a3c04e5a8a290bbe2e20f258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219a6370a3c04e5a8a290bbe2e20f258" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37251 } } }
I20260927 10:45:01.556962 13616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:01.557200 13617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "219a6370a3c04e5a8a290bbe2e20f258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219a6370a3c04e5a8a290bbe2e20f258" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37251 } } }
I20260927 10:45:01.557356 13617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:01.557488 13625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:01.561050 13625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:01.566918 13625 catalog_manager.cc:1347] Generated new cluster ID: 3ca1e5db18c94a2e99d59409afdb646d
I20260927 10:45:01.567054 13625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:01.573997 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:01.568549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.10.96.254:37251)
0927 10:45:01.568647 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:01.568651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:01.568905 (+ 254us) server_negotiation.cc:407] Connection header received
0927 10:45:01.569008 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:01.569011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:01.569063 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:01.569154 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:01.569715 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:01.570541 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:01.571641 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:01.572022 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:01.572126 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:01.573006 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:01.573010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:01.573013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:01.573551 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:01.573584 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:01.573603 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:01.573688 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:01.573792 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 10:45:01.573850 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":35}
I20260927 10:45:01.589737 13625 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:01.591001 13625 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:01.596518 13625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 219a6370a3c04e5a8a290bbe2e20f258: Generated new TSK 0
I20260927 10:45:01.597316 13625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:01.609956 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13542
2026-09-27T10:45:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1407 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:01 dist-test-slave-wmg0 krb5kdc[13643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:01 dist-test-slave-wmg0 krb5kdc[13643](info): set up 2 sockets
Sep 27 10:45:01 dist-test-slave-wmg0 krb5kdc[13643](info): commencing operation
krb5kdc: starting...
W20260927 10:45:02.730086 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 10:45:03.030751 10627 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.301s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 10:45:03.230125 10627 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.199s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 10:45:03.406072 10627 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.176s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/test-user.keytab.
W20260927 10:45:04.151800 10627 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.746s user 0.001s sys 0.001s
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:04.194423 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:39617
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:04.303547 13656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:04.303859 13656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:04.303933 13656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:04.307978 13656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:04.308073 13656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:04.308112 13656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:04.308174 13656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:04.308211 13656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:04.313441 13656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:04.314523 13656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:04.315541 13656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:04.323755 13664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:04.323764 13661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:04.323846 13662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:04.323828 13656 server_base.cc:1034] running on GCE node
I20260927 10:45:04.324553 13656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:04.325734 13656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:04.326898 13656 hybrid_clock.cc:648] HybridClock initialized: now 1790505904326881 us; error 34 us; skew 500 ppm
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:04.329756 13656 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:04.330580 13656 webserver.cc:460] Webserver started at http://127.10.96.254:42229/ using document root <none> and password file <none>
I20260927 10:45:04.331085 13656 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:04.331141 13656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:04.331315 13656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:04.333288 13656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "06d04999619745d19a940920bfb6273e"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "861120726ccbe5f31a96b443a6094280"
server_key_iv: "e925f96f8c8673721225699bd5cfa71a"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.333859 13656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "06d04999619745d19a940920bfb6273e"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "861120726ccbe5f31a96b443a6094280"
server_key_iv: "e925f96f8c8673721225699bd5cfa71a"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.337442 13656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:45:04.339833 13671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.341070 13656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:45:04.341216 13656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "06d04999619745d19a940920bfb6273e"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "861120726ccbe5f31a96b443a6094280"
server_key_iv: "e925f96f8c8673721225699bd5cfa71a"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.341351 13656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:04.372030 13656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:04.375090 13656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:04.375305 13656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:04.382352 13656 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:39617
I20260927 10:45:04.382624 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:39617 every 8 connection(s)
I20260927 10:45:04.383320 13656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:04.386082 13724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:04.390290 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13656
I20260927 10:45:04.390425 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:04.390674 10627 external_mini_cluster.cc:1428] Setting key ac3b0a5846e1cfd930bc9e698c2368aa
I20260927 10:45:04.391605 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e: Bootstrap starting.
I20260927 10:45:04.394017 13724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:04.394908 13724 log.cc:824] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:04.396970 13724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e: No bootstrap required, opened a new log
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:04.401101 13724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d04999619745d19a940920bfb6273e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39617 } }
I20260927 10:45:04.401341 13724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:04.401418 13724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d04999619745d19a940920bfb6273e, State: Initialized, Role: FOLLOWER
I20260927 10:45:04.401911 13724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d04999619745d19a940920bfb6273e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39617 } }
I20260927 10:45:04.402066 13724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:04.402161 13724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:04.402258 13724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:04.403095 13724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d04999619745d19a940920bfb6273e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39617 } }
I20260927 10:45:04.403421 13724 leader_election.cc:302] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d04999619745d19a940920bfb6273e; no voters:
I20260927 10:45:04.403733 13724 leader_election.cc:288] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:04.403802 13729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:04.403999 13729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [term 1 LEADER]: Becoming Leader. State: Replica: 06d04999619745d19a940920bfb6273e, State: Running, Role: LEADER
I20260927 10:45:04.404336 13729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d04999619745d19a940920bfb6273e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39617 } }
I20260927 10:45:04.404981 13724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:04.405967 13731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d04999619745d19a940920bfb6273e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d04999619745d19a940920bfb6273e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39617 } } }
I20260927 10:45:04.406103 13731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:04.406112 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:04.392020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.10.96.254:39617)
0927 10:45:04.392414 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:04.392429 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:04.392463 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:45:04.393174 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:04.393193 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:04.393577 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:04.393857 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:04.394775 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.395963 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.397078 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.397591 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.397705 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:04.400643 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:04.400648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:04.400661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:04.400685 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:04.402668 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:04.403087 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:04.403093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:04.403098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:04.403237 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:04.403520 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:04.403523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:04.403525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:04.404001 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:04.404737 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:04.404869 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 10:45:04.405154 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":135,"threads_started":1}
I20260927 10:45:04.406411 13730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d04999619745d19a940920bfb6273e. Latest consensus state: current_term: 1 leader_uuid: "06d04999619745d19a940920bfb6273e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d04999619745d19a940920bfb6273e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39617 } } }
I20260927 10:45:04.406491 13730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e [sys.catalog]: This master's current role is: LEADER
W20260927 10:45:04.409868 13745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:45:04.409952 13745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:45:04.410030 13738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:04.410733 13738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:04.415889 13738 catalog_manager.cc:1347] Generated new cluster ID: 4bdfac42f6274a6fa5a71ddf0447d144
I20260927 10:45:04.415966 13738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:04.427070 13738 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:04.427919 13738 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:04.438786 13738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d04999619745d19a940920bfb6273e: Generated new TSK 0
I20260927 10:45:04.439457 13738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:45:04.466701 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39617
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:04.574996 13750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:04.575238 13750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:04.575278 13750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:04.579222 13750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:04.579272 13750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:04.579327 13750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:04.584343 13750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:04.585355 13750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:04.586366 13750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:04.595599 13756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:04.595597 13755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:04.595914 13750 server_base.cc:1034] running on GCE node
W20260927 10:45:04.595623 13758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:04.596462 13750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:04.597024 13750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:04.598184 13750 hybrid_clock.cc:648] HybridClock initialized: now 1790505904598168 us; error 26 us; skew 500 ppm
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:04.600723 13750 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:45:04.601580 13750 webserver.cc:460] Webserver started at http://127.10.96.193:46559/ using document root <none> and password file <none>
I20260927 10:45:04.602073 13750 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:04.602128 13750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:04.602308 13750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:04.604172 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "15fa851bce034953b10edd0a9d68ad1c"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "54e0fc9d71ff7502456112b1c8a94311"
server_key_iv: "4626c9e3793ad105db6baaa105226399"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.604673 13750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "15fa851bce034953b10edd0a9d68ad1c"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "54e0fc9d71ff7502456112b1c8a94311"
server_key_iv: "4626c9e3793ad105db6baaa105226399"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.608292 13750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:45:04.610584 13765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.611805 13750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:04.611893 13750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "15fa851bce034953b10edd0a9d68ad1c"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "54e0fc9d71ff7502456112b1c8a94311"
server_key_iv: "4626c9e3793ad105db6baaa105226399"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.611971 13750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:04.635864 13750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:04.639267 13750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:04.639421 13750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:04.639935 13750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:04.640682 13750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:04.640727 13750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.640766 13750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:04.640781 13750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.649921 13750 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:37953
I20260927 10:45:04.649946 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:37953 every 8 connection(s)
I20260927 10:45:04.650804 13750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:04.653169 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13750
I20260927 10:45:04.653318 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:04.653633 10627 external_mini_cluster.cc:1428] Setting key 7ecad6b75bd55f286f4b389be283693b
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:45:04.664951 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:04.652570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:39343 (local address 127.10.96.254:39617)
0927 10:45:04.652723 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:04.652729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:04.653833 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 10:45:04.655423 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:04.655426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:04.655476 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:04.655553 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:04.656679 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.657228 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.658446 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.658678 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.658789 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:04.661190 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:04.661199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:04.661201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:04.661223 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:04.662753 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:04.663336 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:04.663341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:04.663343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:04.663424 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:04.663769 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:04.663772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:04.663774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:04.664002 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:04.664110 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:04.664647 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 10:45:04.664779 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:45:04.665863 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:04.653041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39617 (local address 127.10.96.193:39343)
0927 10:45:04.653639 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:04.653703 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:04.655224 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:04.655727 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:04.655734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:04.656085 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:04.656539 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:04.656552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.657360 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.657364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:04.658322 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:04.658331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.658830 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.658835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:04.659009 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:04.659617 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:04.659632 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:04.661029 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:04.662871 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:04.662878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:04.662893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:04.663234 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:04.663523 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:04.663526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:04.663528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:04.663669 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:04.664107 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:04.664114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:04.664398 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 10:45:04.664608 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 10:45:04.664927 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":129,"threads_started":1}
I20260927 10:45:04.667153 13879 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39617
I20260927 10:45:04.667407 13879 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:04.667893 13879 heartbeater.cc:511] Master 127.10.96.254:39617 requested a full tablet report, sending...
I20260927 10:45:04.669592 13688 ts_manager.cc:194] Registered new tserver with Master: 15fa851bce034953b10edd0a9d68ad1c (127.10.96.193:37953)
I20260927 10:45:04.671216 13688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:39343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:45:04.680065 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39617
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:04.788218 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:04.788465 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:04.788506 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:04.792502 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:04.792557 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:04.792613 13884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:04.797894 13884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:04.798826 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:04.799736 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:04.808974 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:04.808961 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:04.809007 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:04.809307 13884 server_base.cc:1034] running on GCE node
I20260927 10:45:04.809717 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:04.810266 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:04.811455 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790505904811442 us; error 29 us; skew 500 ppm
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:04.814147 13884 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:45:04.815079 13884 webserver.cc:460] Webserver started at http://127.10.96.194:46629/ using document root <none> and password file <none>
I20260927 10:45:04.815694 13884 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:04.815778 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:04.816005 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:04.818028 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "dee0ccd2e5dd485d81dc792cd97b0761"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "2af9b8ee5b40e70eb610f184c2907d64"
server_key_iv: "bf24c86f443bf13d502d939b492b1cb2"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.818594 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "dee0ccd2e5dd485d81dc792cd97b0761"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "2af9b8ee5b40e70eb610f184c2907d64"
server_key_iv: "bf24c86f443bf13d502d939b492b1cb2"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.822126 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:04.824426 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.825728 13884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:04.825857 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "dee0ccd2e5dd485d81dc792cd97b0761"
format_stamp: "Formatted at 2026-09-27 10:45:04 on dist-test-slave-wmg0"
server_key: "2af9b8ee5b40e70eb610f184c2907d64"
server_key_iv: "bf24c86f443bf13d502d939b492b1cb2"
server_key_version: "encryptionkey@0"
I20260927 10:45:04.825963 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:04.849637 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:04.852774 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:04.852995 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:04.853678 13884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:04.854658 13884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:04.854733 13884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.854823 13884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:04.854859 13884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:04.863994 13884 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:45731
I20260927 10:45:04.864012 14012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:45731 every 8 connection(s)
I20260927 10:45:04.864945 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:04.865903 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 13884
I20260927 10:45:04.866046 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:04.866369 10627 external_mini_cluster.cc:1428] Setting key 00d392c4716acd249c3adbaee8ba574e
Sep 27 10:45:04 dist-test-slave-wmg0 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505904, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:45:04.881032 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:04.866847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:46083 (local address 127.10.96.254:39617)
0927 10:45:04.866980 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:04.866984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:04.867940 (+ 956us) server_negotiation.cc:407] Connection header received
0927 10:45:04.869978 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:04.869983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:04.870047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:04.870159 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:04.871478 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.872017 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.873955 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.874177 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.874281 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:04.877120 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:04.877123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:04.877125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:04.877160 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:04.878825 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:04.879391 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:04.879395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:04.879396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:04.879482 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:04.879844 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:04.879849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:04.879850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:04.880041 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:04.880144 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:04.880796 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 10:45:04.880908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:45:04.882177 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:04.867311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39617 (local address 127.10.96.194:46083)
0927 10:45:04.867788 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:04.867830 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:04.869707 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:04.870303 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:04.870310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:04.870661 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:04.871309 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:04.871332 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.872200 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.872206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:04.873817 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:04.873830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:04.874346 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:04.874351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:04.874578 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:04.875195 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:04.875222 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:04.876931 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:04.878954 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:04.878960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:04.878975 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:04.879287 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:04.879581 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:04.879585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:04.879587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:04.879746 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:04.880142 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:04.880149 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:04.880504 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 10:45:04.880739 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 10:45:04.881112 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260927 10:45:04.883473 14013 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39617
I20260927 10:45:04.883744 14013 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:04.884224 14013 heartbeater.cc:511] Master 127.10.96.254:39617 requested a full tablet report, sending...
I20260927 10:45:04.885291 13688 ts_manager.cc:194] Registered new tserver with Master: dee0ccd2e5dd485d81dc792cd97b0761 (127.10.96.194:45731)
I20260927 10:45:04.886138 13688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:46083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:45:04.894261 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39617
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:05.006126 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:05.006485 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:05.006532 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:05.010690 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:05.010743 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:05.010802 14018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:05.015976 14018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:05.016942 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:05.017990 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:05.027190 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:05.027195 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:05.027209 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:05.027247 14018 server_base.cc:1034] running on GCE node
I20260927 10:45:05.027989 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:05.028837 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:05.030011 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790505905029973 us; error 54 us; skew 500 ppm
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:05.032598 14018 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:05.033430 14018 webserver.cc:460] Webserver started at http://127.10.96.195:46217/ using document root <none> and password file <none>
I20260927 10:45:05.034021 14018 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:05.034133 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:05.034425 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:05.037088 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "9a0ab25cf91f4be384de40b5aaccfcbd"
format_stamp: "Formatted at 2026-09-27 10:45:05 on dist-test-slave-wmg0"
server_key: "97aa0ddf97d70d40e7e6baa5da6f0da6"
server_key_iv: "518a404ac27846ead3ee3b20f0dbff99"
server_key_version: "encryptionkey@0"
I20260927 10:45:05.037839 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "9a0ab25cf91f4be384de40b5aaccfcbd"
format_stamp: "Formatted at 2026-09-27 10:45:05 on dist-test-slave-wmg0"
server_key: "97aa0ddf97d70d40e7e6baa5da6f0da6"
server_key_iv: "518a404ac27846ead3ee3b20f0dbff99"
server_key_version: "encryptionkey@0"
I20260927 10:45:05.041391 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:05.043929 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:05.045223 14018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:05.045361 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "9a0ab25cf91f4be384de40b5aaccfcbd"
format_stamp: "Formatted at 2026-09-27 10:45:05 on dist-test-slave-wmg0"
server_key: "97aa0ddf97d70d40e7e6baa5da6f0da6"
server_key_iv: "518a404ac27846ead3ee3b20f0dbff99"
server_key_version: "encryptionkey@0"
I20260927 10:45:05.045506 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:05.066250 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:05.068997 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:05.069191 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:05.069869 14018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:05.070680 14018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:05.070724 14018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:05.070791 14018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:05.070824 14018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:05.079380 14018 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:43365
I20260927 10:45:05.079403 14146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:43365 every 8 connection(s)
I20260927 10:45:05.080298 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505905, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:05.090288 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14018
I20260927 10:45:05.090384 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:05.090731 10627 external_mini_cluster.cc:1428] Setting key bd8027f5bdfd276acdcc908ff045278c
I20260927 10:45:05.094026 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:05.082058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:43237 (local address 127.10.96.254:39617)
0927 10:45:05.082185 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:05.082189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:05.083083 (+ 894us) server_negotiation.cc:407] Connection header received
0927 10:45:05.084436 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:05.084439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:05.084492 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:05.084566 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:05.085756 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:05.086279 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:05.087415 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:05.087636 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:05.087754 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:05.090126 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:05.090129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:05.090132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:05.090164 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:05.091910 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:05.092514 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:05.092517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:05.092518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:05.092600 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:05.093004 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:05.093006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:05.093008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:05.093179 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:05.093256 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:05.093812 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 10:45:05.093911 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:45:05.094836 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:05.082463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39617 (local address 127.10.96.195:43237)
0927 10:45:05.082904 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:05.082957 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:05.084216 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:05.084724 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:05.084732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:05.085075 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:05.085608 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:05.085624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:05.086399 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:05.086404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:05.087275 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:05.087283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:05.087782 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:05.087786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:05.087955 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:05.088495 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:05.088510 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:05.089950 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:05.092026 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:05.092031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:05.092044 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:05.092389 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:05.092732 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:05.092735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:05.092737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:05.092908 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:05.093284 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:05.093289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:05.093558 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 10:45:05.093733 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 10:45:05.093971 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":101,"threads_started":1}
I20260927 10:45:05.095893 14147 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39617
I20260927 10:45:05.096127 14147 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:05.096590 14147 heartbeater.cc:511] Master 127.10.96.254:39617 requested a full tablet report, sending...
I20260927 10:45:05.097594 13688 ts_manager.cc:194] Registered new tserver with Master: 9a0ab25cf91f4be384de40b5aaccfcbd (127.10.96.195:43365)
I20260927 10:45:05.098117 13688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:43237
I20260927 10:45:05.104306 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505905, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505905, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:05.128998 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:05.120650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.10.96.254:39617)
0927 10:45:05.120836 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:05.120842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:05.120978 (+ 136us) server_negotiation.cc:407] Connection header received
0927 10:45:05.121169 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:05.121172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:05.121224 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:05.121318 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:05.122070 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:05.122586 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:05.123419 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:05.123642 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:05.123756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:05.126093 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:05.126096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:05.126098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:05.126122 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:05.127547 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:05.127941 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:05.127944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:05.127945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:05.128018 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:05.128255 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:05.128257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:05.128258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:05.128456 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:05.128559 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:05.128740 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:45:05.128874 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 10:45:05.129452 13688 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59296
I20260927 10:45:05.130344 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13750
I20260927 10:45:05.136533 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13884
I20260927 10:45:05.142198 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14018
I20260927 10:45:05.147362 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 13656
2026-09-27T10:45:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3539 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[14159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[14159](info): set up 2 sockets
Sep 27 10:45:05 dist-test-slave-wmg0 krb5kdc[14159](info): commencing operation
krb5kdc: starting...
W20260927 10:45:06.181681 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:06.298287 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35553
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:06.406584 14172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:06.406819 14172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:06.406883 14172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:06.410790 14172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:06.410840 14172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:06.410857 14172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:06.410874 14172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:06.410918 14172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:06.416049 14172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:35553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:06.417018 14172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:06.418021 14172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:06.426127 14178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:06.426148 14177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:06.426127 14180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:06.426183 14172 server_base.cc:1034] running on GCE node
I20260927 10:45:06.426904 14172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:06.427953 14172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:06.429132 14172 hybrid_clock.cc:648] HybridClock initialized: now 1790505906429105 us; error 41 us; skew 500 ppm
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:06.431775 14172 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:06.432650 14172 webserver.cc:460] Webserver started at http://127.10.96.254:35019/ using document root <none> and password file <none>
I20260927 10:45:06.433188 14172 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:06.433274 14172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:06.433552 14172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:06.435514 14172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "67059fc094fd4d12afde45908342d5fb"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "1ca279610d9f698dd7fd0410fddb4c08"
server_key_iv: "eb7228cdd2197120cf820188edd4a30c"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.436066 14172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "67059fc094fd4d12afde45908342d5fb"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "1ca279610d9f698dd7fd0410fddb4c08"
server_key_iv: "eb7228cdd2197120cf820188edd4a30c"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.439661 14172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 10:45:06.442111 14187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.443289 14172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:45:06.443430 14172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "67059fc094fd4d12afde45908342d5fb"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "1ca279610d9f698dd7fd0410fddb4c08"
server_key_iv: "eb7228cdd2197120cf820188edd4a30c"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.443540 14172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:06.455683 14172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:06.458655 14172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:06.458858 14172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:06.465766 14172 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:35553
I20260927 10:45:06.465775 14239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:35553 every 8 connection(s)
I20260927 10:45:06.466684 14172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:06.469326 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:06.474319 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14172
I20260927 10:45:06.474439 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:06.474701 10627 external_mini_cluster.cc:1428] Setting key 3688534b27b543a7fdd72e3ad7f16622
I20260927 10:45:06.474687 14240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb: Bootstrap starting.
I20260927 10:45:06.477087 14240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:06.477969 14240 log.cc:824] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:06.479820 14240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb: No bootstrap required, opened a new log
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:06.484174 14240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67059fc094fd4d12afde45908342d5fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35553 } }
I20260927 10:45:06.484421 14240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:06.484498 14240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67059fc094fd4d12afde45908342d5fb, State: Initialized, Role: FOLLOWER
I20260927 10:45:06.484977 14240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67059fc094fd4d12afde45908342d5fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35553 } }
I20260927 10:45:06.485126 14240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:06.485211 14240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:06.485312 14240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:06.486259 14240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67059fc094fd4d12afde45908342d5fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35553 } }
I20260927 10:45:06.486601 14240 leader_election.cc:302] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67059fc094fd4d12afde45908342d5fb; no voters:
I20260927 10:45:06.486946 14240 leader_election.cc:288] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:06.487056 14245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:06.487278 14245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [term 1 LEADER]: Becoming Leader. State: Replica: 67059fc094fd4d12afde45908342d5fb, State: Running, Role: LEADER
I20260927 10:45:06.487623 14245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67059fc094fd4d12afde45908342d5fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35553 } }
I20260927 10:45:06.488301 14240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:06.489357 14247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67059fc094fd4d12afde45908342d5fb. Latest consensus state: current_term: 1 leader_uuid: "67059fc094fd4d12afde45908342d5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67059fc094fd4d12afde45908342d5fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35553 } } }
I20260927 10:45:06.489792 14247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:06.489609 14246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67059fc094fd4d12afde45908342d5fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67059fc094fd4d12afde45908342d5fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35553 } } }
I20260927 10:45:06.489719 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:06.475946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.10.96.254:35553)
0927 10:45:06.476378 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:06.476393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:06.476428 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:45:06.477036 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:06.477053 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:06.477531 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:06.477815 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:06.478689 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.479664 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.480795 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.481242 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.481390 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:06.484333 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:06.484338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:06.484351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:06.484378 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:06.486327 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:06.486830 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:06.486837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:06.486847 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:06.486998 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:06.487343 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:06.487347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:06.487348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:06.487830 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:06.488029 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:06.488324 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 10:45:06.488630 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":137,"threads_started":1}
I20260927 10:45:06.490095 14246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb [sys.catalog]: This master's current role is: LEADER
W20260927 10:45:06.493389 14260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:45:06.493486 14260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:45:06.493566 14256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:06.494266 14256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:06.499090 14256 catalog_manager.cc:1347] Generated new cluster ID: e8512abee28b44958ff3170a441cae01
I20260927 10:45:06.499153 14256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:06.536170 14256 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:06.537139 14256 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:06.544525 14256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67059fc094fd4d12afde45908342d5fb: Generated new TSK 0
I20260927 10:45:06.545289 14256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:45:06.576190 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35553
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:06.685578 14266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:06.685881 14266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:06.685959 14266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:06.690271 14266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:06.690351 14266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:06.690448 14266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:06.695711 14266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:06.696795 14266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:06.697911 14266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:06.707176 14271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:06.707198 14272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:06.707366 14266 server_base.cc:1034] running on GCE node
W20260927 10:45:06.707183 14274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:06.707916 14266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:06.708455 14266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:06.709662 14266 hybrid_clock.cc:648] HybridClock initialized: now 1790505906709638 us; error 35 us; skew 500 ppm
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:06.722282 14266 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:45:06.723296 14266 webserver.cc:460] Webserver started at http://127.10.96.193:43063/ using document root <none> and password file <none>
I20260927 10:45:06.723882 14266 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:06.723945 14266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:06.724128 14266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:06.726420 14266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "f4da8a260aed4d10938689272f2d1b98"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "0319006ac668839552731e2372b3903d"
server_key_iv: "a511f998933ac1153f72b4b28561f991"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.726958 14266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "f4da8a260aed4d10938689272f2d1b98"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "0319006ac668839552731e2372b3903d"
server_key_iv: "a511f998933ac1153f72b4b28561f991"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.730860 14266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:45:06.733634 14281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.734850 14266 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:06.734948 14266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "f4da8a260aed4d10938689272f2d1b98"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "0319006ac668839552731e2372b3903d"
server_key_iv: "a511f998933ac1153f72b4b28561f991"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.735030 14266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:06.749641 14266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:06.752921 14266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:06.753073 14266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:06.753614 14266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:06.754356 14266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:06.754405 14266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.754443 14266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:06.754459 14266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.763296 14266 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:40033
I20260927 10:45:06.763360 14394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:40033 every 8 connection(s)
I20260927 10:45:06.764199 14266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:06.772910 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14266
I20260927 10:45:06.773113 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:06.773412 10627 external_mini_cluster.cc:1428] Setting key 29332a40ec42a9bf785934095899ba17
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:06.778189 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:06.766116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:47203 (local address 127.10.96.254:35553)
0927 10:45:06.766275 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:06.766279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:06.767080 (+ 801us) server_negotiation.cc:407] Connection header received
0927 10:45:06.768498 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:06.768502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:06.768560 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:06.768663 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:06.769864 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.770398 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.771540 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.771790 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.771901 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:06.774472 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:06.774476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:06.774478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:06.774510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:06.776065 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:06.776588 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:06.776593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:06.776595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:06.776682 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:06.777045 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:06.777050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:06.777052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:06.777241 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:06.777352 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:06.777909 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 10:45:06.778031 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:45:06.779060 14397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:06.766501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35553 (local address 127.10.96.193:47203)
0927 10:45:06.766932 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:06.766974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:06.768270 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:06.768835 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:06.768841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:06.769204 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:06.769717 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:06.769730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.770527 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.770538 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:06.771413 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:06.771421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.771941 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.771945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:06.772113 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:06.772725 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:06.772742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:06.774310 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:06.776203 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:06.776208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:06.776223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:06.776481 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:06.776784 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:06.776787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:06.776789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:06.776932 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:06.777351 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:06.777357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:06.777674 (+ 317us) client_negotiation.cc:769] Sending connection context
0927 10:45:06.777870 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 10:45:06.778207 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
I20260927 10:45:06.780275 14395 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35553
I20260927 10:45:06.780542 14395 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:06.781124 14395 heartbeater.cc:511] Master 127.10.96.254:35553 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:45:06.782756 14204 ts_manager.cc:194] Registered new tserver with Master: f4da8a260aed4d10938689272f2d1b98 (127.10.96.193:40033)
I20260927 10:45:06.783931 14204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:47203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:45:06.798873 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35553
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:06.906657 14400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:06.906916 14400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:06.906996 14400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:06.911042 14400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:06.911099 14400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:06.911202 14400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:06.916350 14400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:06.917377 14400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:06.918408 14400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:06.927950 14406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:06.928105 14400 server_base.cc:1034] running on GCE node
W20260927 10:45:06.927977 14405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:06.928296 14408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:06.928741 14400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:06.929366 14400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:06.930544 14400 hybrid_clock.cc:648] HybridClock initialized: now 1790505906930524 us; error 30 us; skew 500 ppm
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:06.933291 14400 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:45:06.934265 14400 webserver.cc:460] Webserver started at http://127.10.96.194:46097/ using document root <none> and password file <none>
I20260927 10:45:06.934831 14400 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:06.934892 14400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:06.935132 14400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:06.937130 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "b0d63ec51d0043c886d1638f0d574137"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "565502c41b6ed34344c21a27ad461661"
server_key_iv: "1ee9a1938be65ef7b481f5b78a9833ad"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.937736 14400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "b0d63ec51d0043c886d1638f0d574137"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "565502c41b6ed34344c21a27ad461661"
server_key_iv: "1ee9a1938be65ef7b481f5b78a9833ad"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.941393 14400 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:45:06.943814 14415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.945092 14400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:45:06.945248 14400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "b0d63ec51d0043c886d1638f0d574137"
format_stamp: "Formatted at 2026-09-27 10:45:06 on dist-test-slave-wmg0"
server_key: "565502c41b6ed34344c21a27ad461661"
server_key_iv: "1ee9a1938be65ef7b481f5b78a9833ad"
server_key_version: "encryptionkey@0"
I20260927 10:45:06.945358 14400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:06.967442 14400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:06.970801 14400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:06.970955 14400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:06.971481 14400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:06.972260 14400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:06.972306 14400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.972345 14400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:06.972360 14400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:06.980756 14400 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:36603
I20260927 10:45:06.980845 14528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:36603 every 8 connection(s)
I20260927 10:45:06.981748 14400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:06.984997 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14400
I20260927 10:45:06.985157 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:06.985559 10627 external_mini_cluster.cc:1428] Setting key 7c7f28ee3144f9696ee8300d876c3c4b
Sep 27 10:45:06 dist-test-slave-wmg0 krb5kdc[14159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505906, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:45:06.997442 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:06.983603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:58671 (local address 127.10.96.254:35553)
0927 10:45:06.983753 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:06.983757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:06.984564 (+ 807us) server_negotiation.cc:407] Connection header received
0927 10:45:06.986444 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:06.986448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:06.986502 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:06.986604 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:06.988271 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.988927 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.990599 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.990936 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.991050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:06.993838 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:06.993842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:06.993850 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:06.993876 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:06.995317 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:06.995846 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:06.995849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:06.995850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:06.995926 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:06.996210 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:06.996216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:06.996221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:06.996453 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:06.996537 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:06.997207 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 10:45:06.997317 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:45:06.998381 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:06.983972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35553 (local address 127.10.96.194:58671)
0927 10:45:06.984410 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:06.984450 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:06.986143 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:06.986790 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:06.986798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:06.987363 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:06.988097 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:06.988121 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.989134 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.989147 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:06.990466 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:06.990475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:06.991090 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:06.991094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:06.991273 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:06.991920 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:06.991937 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:06.993668 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:06.995470 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:06.995475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:06.995488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:06.995730 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:06.995987 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:06.995989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:06.995991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:06.996102 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:06.996571 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:06.996577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:06.996843 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 10:45:06.997012 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:45:06.997301 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260927 10:45:06.999810 14529 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35553
I20260927 10:45:07.000072 14529 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:07.000567 14529 heartbeater.cc:511] Master 127.10.96.254:35553 requested a full tablet report, sending...
I20260927 10:45:07.001691 14204 ts_manager.cc:194] Registered new tserver with Master: b0d63ec51d0043c886d1638f0d574137 (127.10.96.194:36603)
I20260927 10:45:07.002557 14204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:58671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:45:07.014032 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35553
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:07.124298 14534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:07.124552 14534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:07.124636 14534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:07.128640 14534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:07.128700 14534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:07.128800 14534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:07.134047 14534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:07.135083 14534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:07.136107 14534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:07.145653 14539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:07.145697 14542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:07.145653 14540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:07.146062 14534 server_base.cc:1034] running on GCE node
I20260927 10:45:07.146476 14534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:07.147040 14534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:07.148227 14534 hybrid_clock.cc:648] HybridClock initialized: now 1790505907148213 us; error 31 us; skew 500 ppm
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505907, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:07.151053 14534 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:07.151832 14534 webserver.cc:460] Webserver started at http://127.10.96.195:42679/ using document root <none> and password file <none>
I20260927 10:45:07.152335 14534 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:07.152388 14534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:07.152544 14534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:07.154520 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "b96f41cc731c40069a29f658d1116f04"
format_stamp: "Formatted at 2026-09-27 10:45:07 on dist-test-slave-wmg0"
server_key: "2ddb8a14e30cdd170b38b32bd5cea293"
server_key_iv: "ef5332da8aa1d3516c452f3162bb76ef"
server_key_version: "encryptionkey@0"
I20260927 10:45:07.155014 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "b96f41cc731c40069a29f658d1116f04"
format_stamp: "Formatted at 2026-09-27 10:45:07 on dist-test-slave-wmg0"
server_key: "2ddb8a14e30cdd170b38b32bd5cea293"
server_key_iv: "ef5332da8aa1d3516c452f3162bb76ef"
server_key_version: "encryptionkey@0"
I20260927 10:45:07.158550 14534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:45:07.160816 14549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:07.162066 14534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:07.162184 14534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "b96f41cc731c40069a29f658d1116f04"
format_stamp: "Formatted at 2026-09-27 10:45:07 on dist-test-slave-wmg0"
server_key: "2ddb8a14e30cdd170b38b32bd5cea293"
server_key_iv: "ef5332da8aa1d3516c452f3162bb76ef"
server_key_version: "encryptionkey@0"
I20260927 10:45:07.162302 14534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:07.182283 14534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:07.185699 14534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:07.185904 14534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:07.186534 14534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:07.187353 14534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:07.187399 14534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:07.187471 14534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:07.187510 14534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:07.196118 14534 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:34285
I20260927 10:45:07.196143 14662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:34285 every 8 connection(s)
I20260927 10:45:07.196964 14534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:07.200419 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14534
I20260927 10:45:07.200564 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:07.200852 10627 external_mini_cluster.cc:1428] Setting key 07f1a03ec926f73d21129901ffe488b9
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505907, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:07.210150 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:07.198727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56951 (local address 127.10.96.254:35553)
0927 10:45:07.198871 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:07.198876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:07.199788 (+ 912us) server_negotiation.cc:407] Connection header received
0927 10:45:07.201248 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:07.201251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:07.201308 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:07.201408 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:07.202544 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.203060 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.204164 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.204394 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.204503 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:07.206874 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:07.206879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:07.206886 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:07.206909 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:07.208174 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:07.208651 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:07.208655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:07.208657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:07.208739 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:07.209056 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:07.209061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:07.209063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:07.209256 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:07.209366 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:07.209880 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 10:45:07.209990 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:45:07.210889 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:07.199198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35553 (local address 127.10.96.195:56951)
0927 10:45:07.199640 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:07.199682 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:07.201013 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:07.201607 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:07.201613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:07.201957 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:07.202408 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:07.202421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.203179 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.203182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:07.204047 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:07.204055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.204533 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.204537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:07.204710 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:07.205235 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:07.205252 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:07.206715 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:07.208293 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:07.208300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:07.208314 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:07.208553 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:07.208836 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:07.208839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:07.208841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:07.208966 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:07.209361 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:07.209367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:07.209655 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 10:45:07.209843 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 10:45:07.210131 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260927 10:45:07.211951 14663 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35553
I20260927 10:45:07.212181 14663 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:07.212630 14663 heartbeater.cc:511] Master 127.10.96.254:35553 requested a full tablet report, sending...
I20260927 10:45:07.213757 14203 ts_manager.cc:194] Registered new tserver with Master: b96f41cc731c40069a29f658d1116f04 (127.10.96.195:34285)
I20260927 10:45:07.214411 14203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:56951
I20260927 10:45:07.214579 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:45:07.240058 14670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:07.230689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.10.96.193:40033)
0927 10:45:07.231018 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:07.231034 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:07.231063 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:45:07.231121 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:07.231135 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:07.231324 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:07.231479 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:07.232159 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.232957 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.233853 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.234236 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.234350 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:07.236679 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:07.236689 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:07.236706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:07.236744 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:07.238300 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:07.238731 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:07.238737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:07.238739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:07.238817 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:07.239089 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:07.239094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:07.239096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:07.239340 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:07.239559 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:07.239737 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 10:45:07.239890 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":119,"threads_started":1}
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:07.250983 14243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:07.243070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.10.96.254:35553)
0927 10:45:07.243205 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:07.243209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:07.243222 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:07.243308 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:07.243310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:07.243349 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:07.243417 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:07.243987 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.244516 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.245254 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:07.245518 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:07.245607 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:07.247886 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:07.247889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:07.247892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:07.247916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:07.249380 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:07.249873 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:07.249876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:07.249877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:07.249954 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:07.250181 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:07.250183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:07.250184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:07.250417 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:07.250527 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:07.250695 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 10:45:07.250789 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 10:45:07.251395 14203 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48614
I20260927 10:45:07.252020 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14266
I20260927 10:45:07.257402 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14400
I20260927 10:45:07.263083 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14534
I20260927 10:45:07.269948 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14172
2026-09-27T10:45:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2123 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14675](info): set up 2 sockets
Sep 27 10:45:07 dist-test-slave-wmg0 krb5kdc[14675](info): commencing operation
krb5kdc: starting...
W20260927 10:45:08.305810 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:08 dist-test-slave-wmg0 krb5kdc[14675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:08.424476 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35341
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:08.535135 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:08.535415 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:08.535459 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:08.539844 14688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:08.539903 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:08.539922 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:08.539940 14688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:08.539955 14688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:08.545188 14688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:08.546264 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:08.547312 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:08.555414 14696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:08.555403 14693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:08.555418 14694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:08.555816 14688 server_base.cc:1034] running on GCE node
I20260927 10:45:08.556295 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:08.557292 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:08.558490 14688 hybrid_clock.cc:648] HybridClock initialized: now 1790505908558478 us; error 33 us; skew 500 ppm
Sep 27 10:45:08 dist-test-slave-wmg0 krb5kdc[14675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:08.561203 14688 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:08.561959 14688 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:08.562053 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:08.562278 14688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:08.564275 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "1999e1624d614c10950a8f8d18d22244"
format_stamp: "Formatted at 2026-09-27 10:45:08 on dist-test-slave-wmg0"
server_key: "efdb24b8a1ade3427d0ba366e0d8b3fd"
server_key_iv: "ab9e83e61158a8ed3b52ffd44c3f3dd9"
server_key_version: "encryptionkey@0"
I20260927 10:45:08.564859 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "1999e1624d614c10950a8f8d18d22244"
format_stamp: "Formatted at 2026-09-27 10:45:08 on dist-test-slave-wmg0"
server_key: "efdb24b8a1ade3427d0ba366e0d8b3fd"
server_key_iv: "ab9e83e61158a8ed3b52ffd44c3f3dd9"
server_key_version: "encryptionkey@0"
I20260927 10:45:08.568454 14688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:45:08.570936 14702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:08.572157 14688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:08.572307 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "1999e1624d614c10950a8f8d18d22244"
format_stamp: "Formatted at 2026-09-27 10:45:08 on dist-test-slave-wmg0"
server_key: "efdb24b8a1ade3427d0ba366e0d8b3fd"
server_key_iv: "ab9e83e61158a8ed3b52ffd44c3f3dd9"
server_key_version: "encryptionkey@0"
I20260927 10:45:08.572419 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:08.584540 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:08.587738 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:08.587944 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:08.594993 14688 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:35341
I20260927 10:45:08.595263 14754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:35341 every 8 connection(s)
I20260927 10:45:08.595679 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:08.598500 14755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:08.600477 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14688
I20260927 10:45:08.600598 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:08.600880 10627 external_mini_cluster.cc:1428] Setting key c5f10e928b87c9685721894ccaf299d7
I20260927 10:45:08.605371 14755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244: Bootstrap starting.
I20260927 10:45:08.607882 14755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:08.608786 14755 log.cc:824] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:45:08 dist-test-slave-wmg0 krb5kdc[14675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:08.611001 14755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244: No bootstrap required, opened a new log
I20260927 10:45:08.615687 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:08.602307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.10.96.254:35341)
0927 10:45:08.602754 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:08.602771 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:08.602807 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:45:08.603547 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:08.603564 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:08.603887 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:08.604177 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:08.605049 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:08.606126 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:08.607290 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:08.607731 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:08.607850 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:08.610551 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:08.610557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:08.610573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:08.610608 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:08.612561 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:08.612967 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:08.612973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:08.612979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:08.613125 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:08.613393 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:08.613396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:08.613399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:08.613971 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:08.614155 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:08.614404 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 10:45:08.614702 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":137,"threads_started":1}
I20260927 10:45:08.615877 14755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1999e1624d614c10950a8f8d18d22244" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35341 } }
I20260927 10:45:08.616137 14755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:08.616192 14755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1999e1624d614c10950a8f8d18d22244, State: Initialized, Role: FOLLOWER
I20260927 10:45:08.616623 14755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1999e1624d614c10950a8f8d18d22244" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35341 } }
I20260927 10:45:08.616786 14755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:08.616899 14755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:08.617019 14755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:08.618079 14755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1999e1624d614c10950a8f8d18d22244" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35341 } }
I20260927 10:45:08.618419 14755 leader_election.cc:302] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1999e1624d614c10950a8f8d18d22244; no voters:
I20260927 10:45:08.618772 14755 leader_election.cc:288] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:08.618884 14760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:08.619117 14760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [term 1 LEADER]: Becoming Leader. State: Replica: 1999e1624d614c10950a8f8d18d22244, State: Running, Role: LEADER
I20260927 10:45:08.619514 14760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1999e1624d614c10950a8f8d18d22244" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35341 } }
I20260927 10:45:08.620000 14755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:08.621508 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1999e1624d614c10950a8f8d18d22244. Latest consensus state: current_term: 1 leader_uuid: "1999e1624d614c10950a8f8d18d22244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1999e1624d614c10950a8f8d18d22244" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35341 } } }
I20260927 10:45:08.621556 14761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1999e1624d614c10950a8f8d18d22244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1999e1624d614c10950a8f8d18d22244" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35341 } } }
I20260927 10:45:08.621762 14761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:08.622140 14769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:08.622515 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:08.624949 14769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:08.629953 14769 catalog_manager.cc:1347] Generated new cluster ID: 494bf31d8bb540bb9412e1776154b695
I20260927 10:45:08.630014 14769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:08.648057 14769 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:08.649410 14769 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:08.664669 14769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244: Generated new TSK 0
I20260927 10:45:08.665650 14769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:45:08.696074 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35341
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:08.806352 14781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:08.806612 14781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:08.806658 14781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:08.810672 14781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:08.810732 14781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:08.810820 14781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:08.815994 14781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:08.817082 14781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:08.818215 14781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:45:08.828752 14781 server_base.cc:1034] running on GCE node
W20260927 10:45:08.828783 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:08.828933 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:08.828781 14789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:08.829440 14781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:08.830128 14781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:08.831300 14781 hybrid_clock.cc:648] HybridClock initialized: now 1790505908831285 us; error 29 us; skew 500 ppm
Sep 27 10:45:08 dist-test-slave-wmg0 krb5kdc[14675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:08.834120 14781 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:45:08.834848 14781 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:08.834908 14781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:08.835121 14781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:08.837146 14781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "41a538e5b7354916ad56596264facb1f"
format_stamp: "Formatted at 2026-09-27 10:45:08 on dist-test-slave-wmg0"
server_key: "f17c6b2308fe3af9605e1870753c04bc"
server_key_iv: "d0bf4ecca48f72b8d46bc8ffeb01fa42"
server_key_version: "encryptionkey@0"
I20260927 10:45:08.837743 14781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "41a538e5b7354916ad56596264facb1f"
format_stamp: "Formatted at 2026-09-27 10:45:08 on dist-test-slave-wmg0"
server_key: "f17c6b2308fe3af9605e1870753c04bc"
server_key_iv: "d0bf4ecca48f72b8d46bc8ffeb01fa42"
server_key_version: "encryptionkey@0"
I20260927 10:45:08.841354 14781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 10:45:08.843714 14795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:08.844925 14781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:08.845058 14781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "41a538e5b7354916ad56596264facb1f"
format_stamp: "Formatted at 2026-09-27 10:45:08 on dist-test-slave-wmg0"
server_key: "f17c6b2308fe3af9605e1870753c04bc"
server_key_iv: "d0bf4ecca48f72b8d46bc8ffeb01fa42"
server_key_version: "encryptionkey@0"
I20260927 10:45:08.845167 14781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:08.863463 14781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:08.866791 14781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:08.867004 14781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:08.867589 14781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:08.868414 14781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:08.868486 14781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:08.868562 14781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:08.868613 14781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:08.877517 14781 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:41029
I20260927 10:45:08.877533 14908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:41029 every 8 connection(s)
I20260927 10:45:08.878082 14781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:08.882187 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14781
I20260927 10:45:08.882308 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:08.882597 10627 external_mini_cluster.cc:1428] Setting key db56410922d410d34a74325a5f162e96
Sep 27 10:45:08 dist-test-slave-wmg0 krb5kdc[14675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505908, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:45:08.893162 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:08.879703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:45485 (local address 127.10.96.254:35341)
0927 10:45:08.879868 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:08.879873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:08.881041 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 10:45:08.881948 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:08.881951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:08.881997 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:08.882073 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:08.883455 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:08.884177 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:08.885970 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:08.886376 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:08.886510 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:08.889250 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:08.889253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:08.889255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:08.889278 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:08.890890 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:08.891504 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:08.891506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:08.891507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:08.891581 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:08.891978 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:08.891980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:08.891982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:08.892174 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:08.892254 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:08.892918 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 10:45:08.893040 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:45:08.894127 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:08.880094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35341 (local address 127.10.96.193:45485)
0927 10:45:08.880869 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:08.880908 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:08.881727 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:08.882297 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:08.882307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:08.882784 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:08.883321 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:08.883334 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:08.884326 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:08.884330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:08.885826 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:08.885842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:08.886527 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:08.886530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:08.886668 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:08.887282 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:08.887298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:08.889063 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:08.891009 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:08.891015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:08.891031 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:08.891405 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:08.891675 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:08.891680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:08.891685 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:08.891874 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:08.892283 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:08.892292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:08.892616 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 10:45:08.892845 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 10:45:08.893247 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:45:08.895464 14909 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35341
I20260927 10:45:08.895699 14909 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:08.896164 14909 heartbeater.cc:511] Master 127.10.96.254:35341 requested a full tablet report, sending...
I20260927 10:45:08.897679 14719 ts_manager.cc:194] Registered new tserver with Master: 41a538e5b7354916ad56596264facb1f (127.10.96.193:41029)
I20260927 10:45:08.898761 14719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:45485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:45:08.910534 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35341
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:09.018267 14914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:09.018517 14914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:09.018560 14914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:09.022820 14914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:09.022876 14914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:09.022958 14914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:09.028069 14914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:09.029081 14914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:09.030083 14914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:09.039402 14920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.039495 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.039404 14919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:09.039479 14914 server_base.cc:1034] running on GCE node
I20260927 10:45:09.040017 14914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:09.040622 14914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:09.041826 14914 hybrid_clock.cc:648] HybridClock initialized: now 1790505909041808 us; error 28 us; skew 500 ppm
Sep 27 10:45:09 dist-test-slave-wmg0 krb5kdc[14675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505909, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:09.044616 14914 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:45:09.045374 14914 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:09.045463 14914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:09.045718 14914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:09.047854 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "e81dc95df30f4d83a6eeb70fc173d275"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "a74ec6ea466792e9375ff772ea835c04"
server_key_iv: "e0ff6f64d79390edc20d8ccf650ee6a8"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.048447 14914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "e81dc95df30f4d83a6eeb70fc173d275"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "a74ec6ea466792e9375ff772ea835c04"
server_key_iv: "e0ff6f64d79390edc20d8ccf650ee6a8"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.053018 14914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 10:45:09.055698 14928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.056945 14914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:09.057083 14914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "e81dc95df30f4d83a6eeb70fc173d275"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "a74ec6ea466792e9375ff772ea835c04"
server_key_iv: "e0ff6f64d79390edc20d8ccf650ee6a8"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.057222 14914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:09.083063 14914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:09.086316 14914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:09.086544 14914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:09.087141 14914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:09.087980 14914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:09.088054 14914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.088129 14914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:09.088173 14914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.097244 14914 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:39965
I20260927 10:45:09.097260 15041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:39965 every 8 connection(s)
I20260927 10:45:09.097852 14914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:09.106652 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 14914
I20260927 10:45:09.106757 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:09.107053 10627 external_mini_cluster.cc:1428] Setting key 8d64ecc06c4db8c31d75dd58c0a9762e
Sep 27 10:45:09 dist-test-slave-wmg0 krb5kdc[14675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505909, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:09.111961 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.099622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36487 (local address 127.10.96.254:35341)
0927 10:45:09.099786 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.099791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.100881 (+ 1090us) server_negotiation.cc:407] Connection header received
0927 10:45:09.101799 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.101802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.101845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.101909 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:09.103300 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.103940 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.105021 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.105236 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.105314 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.108056 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:09.108059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:09.108061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:09.108084 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:09.109910 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:09.110420 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:09.110427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:09.110429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:09.110516 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:09.110875 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:09.110878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:09.110880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:09.111066 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:09.111175 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.111671 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.111799 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 10:45:09.112794 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.099952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35341 (local address 127.10.96.194:36487)
0927 10:45:09.100731 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.100779 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.101603 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.102064 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.102071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.102436 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:09.103098 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.103112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.104045 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.104048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.104917 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.104925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.105385 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.105388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.105615 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.106222 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:09.106238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:09.107872 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:09.110022 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:09.110026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:09.110039 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:09.110292 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:09.110609 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:09.110614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:09.110616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:09.110757 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:09.111170 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:09.111177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:09.111434 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.111619 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.111962 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":98,"threads_started":1}
I20260927 10:45:09.113988 15042 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35341
I20260927 10:45:09.114264 15042 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:09.114845 15042 heartbeater.cc:511] Master 127.10.96.254:35341 requested a full tablet report, sending...
I20260927 10:45:09.116225 14719 ts_manager.cc:194] Registered new tserver with Master: e81dc95df30f4d83a6eeb70fc173d275 (127.10.96.194:39965)
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:45:09.117043 14719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:36487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:45:09.134088 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35341
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:09.243997 15047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:09.244278 15047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:09.244324 15047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:09.248471 15047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:09.248553 15047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:09.248656 15047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:09.253978 15047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:09.255016 15047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:09.256028 15047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:09.266268 15055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.266263 15052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.266263 15053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:09.266885 15047 server_base.cc:1034] running on GCE node
I20260927 10:45:09.267393 15047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:09.268273 15047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:09.269512 15047 hybrid_clock.cc:648] HybridClock initialized: now 1790505909269493 us; error 33 us; skew 500 ppm
Sep 27 10:45:09 dist-test-slave-wmg0 krb5kdc[14675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505909, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:09.272161 15047 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:09.272939 15047 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:09.273026 15047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:09.273239 15047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:09.275322 15047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "8f0f56bbd683084fa46e37f8af375bf9"
server_key_iv: "0ae93512b7bd9ec53bef1b3aad105372"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.275945 15047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "8f0f56bbd683084fa46e37f8af375bf9"
server_key_iv: "0ae93512b7bd9ec53bef1b3aad105372"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.280474 15047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:45:09.282898 15061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.284206 15047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:09.284359 15047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "8f0f56bbd683084fa46e37f8af375bf9"
server_key_iv: "0ae93512b7bd9ec53bef1b3aad105372"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.284474 15047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:09.293794 15047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:09.296832 15047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:09.297056 15047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:09.297760 15047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:09.298581 15047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:09.298660 15047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.298728 15047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:09.298780 15047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.307615 15047 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:41587
I20260927 10:45:09.307637 15174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:41587 every 8 connection(s)
I20260927 10:45:09.308190 15047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:09.309996 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15047
I20260927 10:45:09.310133 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:09.310410 10627 external_mini_cluster.cc:1428] Setting key a5257c91fca922658e441dd2851d71d3
Sep 27 10:45:09 dist-test-slave-wmg0 krb5kdc[14675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505909, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:09.321609 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.309998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:45319 (local address 127.10.96.254:35341)
0927 10:45:09.310174 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.310181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.311104 (+ 923us) server_negotiation.cc:407] Connection header received
0927 10:45:09.312601 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.312604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.312645 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.312730 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:09.313886 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.314397 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.315502 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.315710 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.315806 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.318170 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:09.318173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:09.318175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:09.318198 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:09.319676 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:09.320123 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:09.320126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:09.320127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:09.320263 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:09.320551 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:09.320553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:09.320554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:09.320725 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:09.320869 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.321320 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.321445 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:45:09.322515 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.310418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35341 (local address 127.10.96.195:45319)
0927 10:45:09.310931 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.310988 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.312379 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.312868 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.312875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.313216 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:09.313762 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.313776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.314498 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.314500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.315394 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.315402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.315848 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.315850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.316000 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.316541 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:09.316555 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:09.318028 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:09.319780 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:09.319784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:09.319795 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:09.320027 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:09.320338 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:09.320340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:09.320342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:09.320468 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:09.320865 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:09.320871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:09.321086 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.321282 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.321626 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260927 10:45:09.323650 15175 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35341
I20260927 10:45:09.323925 15175 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:09.324425 15175 heartbeater.cc:511] Master 127.10.96.254:35341 requested a full tablet report, sending...
I20260927 10:45:09.325388 14718 ts_manager.cc:194] Registered new tserver with Master: e95ec4fcb56c45d79b2635a5fa28d5f7 (127.10.96.195:41587)
I20260927 10:45:09.325984 14718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:45319
I20260927 10:45:09.336005 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:45:09.345700 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.338044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.10.96.254:35341)
0927 10:45:09.338190 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.338195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.338350 (+ 155us) server_negotiation.cc:407] Connection header received
0927 10:45:09.338505 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.338508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.338554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.338638 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:09.339333 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.339854 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.340794 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.341009 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.341117 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.342472 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:09.342476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:09.342478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:09.342502 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:09.343899 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:09.344296 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:09.344298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:09.344300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:09.344374 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:09.344690 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:09.344693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:09.344694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:09.344909 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:09.345021 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.345314 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.345475 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:09.348628 14718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:09.351871 14718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:09.365847 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.360626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.10.96.193:41029)
0927 10:45:09.361155 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.361167 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.361186 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:45:09.361289 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.361296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.361528 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.362026 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:09.362169 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.363243 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.364564 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.365117 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.365240 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.365394 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.365599 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.365694 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":62,"threads_started":1}
I20260927 10:45:09.365846 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.359801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:41029 (local address 127.0.0.1:33702)
0927 10:45:09.360536 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.360583 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.360711 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.361732 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.361735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.361762 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:09.361869 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.361876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.363373 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.363376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.364452 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.364458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.365237 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.365241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.365343 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.365378 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.365562 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.365684 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260927 10:45:09.366166 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.360031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53758 (local address 127.10.96.194:39965)
0927 10:45:09.361004 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.361015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.361207 (+ 192us) server_negotiation.cc:407] Connection header received
0927 10:45:09.361292 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.361299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.361557 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.361724 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:09.362117 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.362820 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.364365 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.365176 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.365288 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.365417 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.365914 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.366012 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":88,"threads_started":1}
I20260927 10:45:09.366226 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.360041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:41587 (local address 127.0.0.1:36938)
0927 10:45:09.361145 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.361158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.361245 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.361699 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.361704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.361726 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:09.361888 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.361895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.363713 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.363716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.364658 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.364668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.365871 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.365875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.365953 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.365973 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.366040 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.366101 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":7,"thread_start_us":91,"threads_started":1}
I20260927 10:45:09.366643 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.359810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39965 (local address 127.0.0.1:53758)
0927 10:45:09.361013 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.361027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.361112 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.361882 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.361885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.361900 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:09.361999 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.362005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.362946 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.362951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.364235 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.364245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.365517 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.365520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.365625 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.365661 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.365723 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.365793 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260927 10:45:09.366775 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.360569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.10.96.195:41587)
0927 10:45:09.360988 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.361002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.361243 (+ 241us) server_negotiation.cc:407] Connection header received
0927 10:45:09.361337 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.361341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.361528 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.361681 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:09.362002 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.363572 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.364800 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.365579 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.365673 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.365813 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.366530 (+ 717us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.366627 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":151,"threads_started":1}
I20260927 10:45:09.369001 14843 tablet_service.cc:1467] Processing CreateTablet for tablet 087e45ac11fa43f7a756c722033d184b (DEFAULT_TABLE table=test-table [id=f0ef90864fbd42718d33e7422d35aa21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:09.369390 15109 tablet_service.cc:1467] Processing CreateTablet for tablet 087e45ac11fa43f7a756c722033d184b (DEFAULT_TABLE table=test-table [id=f0ef90864fbd42718d33e7422d35aa21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:09.369391 14976 tablet_service.cc:1467] Processing CreateTablet for tablet 087e45ac11fa43f7a756c722033d184b (DEFAULT_TABLE table=test-table [id=f0ef90864fbd42718d33e7422d35aa21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:09.369984 14843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087e45ac11fa43f7a756c722033d184b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:09.370381 14976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087e45ac11fa43f7a756c722033d184b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:09.370388 15109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087e45ac11fa43f7a756c722033d184b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:09.375535 15192 tablet_bootstrap.cc:492] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: Bootstrap starting.
I20260927 10:45:09.377885 15193 tablet_bootstrap.cc:492] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Bootstrap starting.
I20260927 10:45:09.377931 15192 tablet_bootstrap.cc:654] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:09.378932 15192 log.cc:824] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:09.379511 15194 tablet_bootstrap.cc:492] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: Bootstrap starting.
I20260927 10:45:09.379599 15193 tablet_bootstrap.cc:654] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:09.380760 15193 log.cc:824] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:09.380990 15192 tablet_bootstrap.cc:492] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: No bootstrap required, opened a new log
I20260927 10:45:09.381188 15192 ts_tablet_manager.cc:1397] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:45:09.381438 15194 tablet_bootstrap.cc:654] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:09.382261 15194 log.cc:824] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:09.382643 15193 tablet_bootstrap.cc:492] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: No bootstrap required, opened a new log
I20260927 10:45:09.382827 15193 ts_tablet_manager.cc:1397] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:45:09.383863 15194 tablet_bootstrap.cc:492] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: No bootstrap required, opened a new log
I20260927 10:45:09.384049 15194 ts_tablet_manager.cc:1397] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:45:09.385037 15192 raft_consensus.cc:348] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.385316 15192 raft_consensus.cc:374] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:09.385375 15192 raft_consensus.cc:729] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a538e5b7354916ad56596264facb1f, State: Initialized, Role: FOLLOWER
I20260927 10:45:09.385831 15192 consensus_queue.cc:260] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.386462 14909 heartbeater.cc:503] Master 127.10.96.254:35341 was elected leader, sending a full tablet report...
I20260927 10:45:09.386430 15193 raft_consensus.cc:348] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.386683 15193 raft_consensus.cc:374] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:09.386745 15193 raft_consensus.cc:729] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95ec4fcb56c45d79b2635a5fa28d5f7, State: Initialized, Role: FOLLOWER
I20260927 10:45:09.387142 15193 consensus_queue.cc:260] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.387275 15194 raft_consensus.cc:348] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.387524 15194 raft_consensus.cc:374] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:09.387585 15194 raft_consensus.cc:729] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81dc95df30f4d83a6eeb70fc173d275, State: Initialized, Role: FOLLOWER
I20260927 10:45:09.388000 15194 consensus_queue.cc:260] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.388095 15193 ts_tablet_manager.cc:1428] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:45:09.388095 15192 ts_tablet_manager.cc:1428] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 10:45:09.388397 15175 heartbeater.cc:503] Master 127.10.96.254:35341 was elected leader, sending a full tablet report...
I20260927 10:45:09.388926 15194 ts_tablet_manager.cc:1428] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:45:09.389271 15042 heartbeater.cc:503] Master 127.10.96.254:35341 was elected leader, sending a full tablet report...
I20260927 10:45:09.502897 15200 raft_consensus.cc:482] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:09.503129 15200 raft_consensus.cc:504] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.504413 15200 leader_election.cc:288] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41a538e5b7354916ad56596264facb1f (127.10.96.193:41029), e95ec4fcb56c45d79b2635a5fa28d5f7 (127.10.96.195:41587)
I20260927 10:45:09.508000 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.504684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:41587 (local address 127.10.96.194:49829)
0927 10:45:09.504822 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.504840 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.504949 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.505252 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.505255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.505273 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:09.505397 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.505402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.506157 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.506163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.506952 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.506960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.507583 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.507586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.507688 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.507709 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.507802 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.507869 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260927 10:45:09.508000 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.504892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:49829 (local address 127.10.96.195:41587)
0927 10:45:09.505036 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.505041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.505054 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:09.505107 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.505109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.505144 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.505233 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:09.505533 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.506048 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.507074 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.507465 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.507593 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.507643 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.507828 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.507892 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:45:09.508376 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.504793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:50979 (local address 127.10.96.193:41029)
0927 10:45:09.504944 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.504951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.505386 (+ 435us) server_negotiation.cc:407] Connection header received
0927 10:45:09.505577 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.505580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.505628 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.505710 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:09.505961 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.506444 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.507448 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.507827 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.507950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.507995 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.508195 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.508259 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:45:09.508705 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.504684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:41029 (local address 127.10.96.194:50979)
0927 10:45:09.505287 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.505299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:09.505422 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:09.505720 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:09.505723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:09.505737 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:09.505853 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.505858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.506552 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.506556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.507326 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:09.507334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.507940 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.507943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:09.508038 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.508065 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:45:09.508537 (+ 472us) client_negotiation.cc:240] Negotiation successful
0927 10:45:09.508605 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":263,"thread_start_us":146,"threads_started":1}
I20260927 10:45:09.508670 15129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087e45ac11fa43f7a756c722033d184b" candidate_uuid: "e81dc95df30f4d83a6eeb70fc173d275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" is_pre_election: true
I20260927 10:45:09.508992 15129 raft_consensus.cc:2454] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e81dc95df30f4d83a6eeb70fc173d275 in term 0.
I20260927 10:45:09.509343 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087e45ac11fa43f7a756c722033d184b" candidate_uuid: "e81dc95df30f4d83a6eeb70fc173d275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a538e5b7354916ad56596264facb1f" is_pre_election: true
I20260927 10:45:09.509519 14932 leader_election.cc:302] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e81dc95df30f4d83a6eeb70fc173d275, e95ec4fcb56c45d79b2635a5fa28d5f7; no voters:
I20260927 10:45:09.509648 14863 raft_consensus.cc:2454] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e81dc95df30f4d83a6eeb70fc173d275 in term 0.
I20260927 10:45:09.509800 15200 raft_consensus.cc:2781] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:09.509883 15200 raft_consensus.cc:482] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:09.510092 15200 raft_consensus.cc:3037] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:09.511129 15200 raft_consensus.cc:504] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.511509 15200 leader_election.cc:288] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [CANDIDATE]: Term 1 election: Requested vote from peers 41a538e5b7354916ad56596264facb1f (127.10.96.193:41029), e95ec4fcb56c45d79b2635a5fa28d5f7 (127.10.96.195:41587)
I20260927 10:45:09.511869 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087e45ac11fa43f7a756c722033d184b" candidate_uuid: "e81dc95df30f4d83a6eeb70fc173d275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a538e5b7354916ad56596264facb1f"
I20260927 10:45:09.511889 15129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087e45ac11fa43f7a756c722033d184b" candidate_uuid: "e81dc95df30f4d83a6eeb70fc173d275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7"
I20260927 10:45:09.512025 14863 raft_consensus.cc:3037] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:09.512042 15129 raft_consensus.cc:3037] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:09.513033 14863 raft_consensus.cc:2454] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e81dc95df30f4d83a6eeb70fc173d275 in term 1.
I20260927 10:45:09.513092 15129 raft_consensus.cc:2454] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e81dc95df30f4d83a6eeb70fc173d275 in term 1.
I20260927 10:45:09.513360 14929 leader_election.cc:302] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a538e5b7354916ad56596264facb1f, e81dc95df30f4d83a6eeb70fc173d275; no voters:
I20260927 10:45:09.513614 15200 raft_consensus.cc:2781] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:09.513789 15200 raft_consensus.cc:686] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 1 LEADER]: Becoming Leader. State: Replica: e81dc95df30f4d83a6eeb70fc173d275, State: Running, Role: LEADER
I20260927 10:45:09.514106 15200 consensus_queue.cc:237] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } }
I20260927 10:45:09.517403 14719 catalog_manager.cc:5495] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 reported cstate change: term changed from 0 to 1, leader changed from <none> to e81dc95df30f4d83a6eeb70fc173d275 (127.10.96.194). New cstate: current_term: 1 leader_uuid: "e81dc95df30f4d83a6eeb70fc173d275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e81dc95df30f4d83a6eeb70fc173d275" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 } health_report { overall_health: UNKNOWN } } }
W20260927 10:45:09.559042 15176 tablet.cc:2369] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:45:09.561611 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.557866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.10.96.194:39965)
0927 10:45:09.557991 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.558000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.558017 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:45:09.558177 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.558179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.558233 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.558322 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:09.558532 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.559033 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.560034 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:09.560254 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:09.560337 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:09.560676 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:09.561199 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:09.561293 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.561397 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.561450 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:45:09.567260 15129 raft_consensus.cc:1261] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 1 FOLLOWER]: Refusing update from remote peer e81dc95df30f4d83a6eeb70fc173d275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:09.567262 14863 raft_consensus.cc:1261] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 1 FOLLOWER]: Refusing update from remote peer e81dc95df30f4d83a6eeb70fc173d275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:09.568034 15202 consensus_queue.cc:1035] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95ec4fcb56c45d79b2635a5fa28d5f7" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:09.568221 15200 consensus_queue.cc:1035] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a538e5b7354916ad56596264facb1f" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 41029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:09.576352 15207 mvcc.cc:205] Tried to move back new op lower bound from 7333912205580451840 to 7333912205372428288. Current Snapshot: MvccSnapshot[applied={T|T < 7333912205580451840}]
I20260927 10:45:09.583722 14717 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:09.583914 14717 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:09.586387 14717 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1999e1624d614c10950a8f8d18d22244: Sending DeleteTablet for 3 replicas of tablet 087e45ac11fa43f7a756c722033d184b
I20260927 10:45:09.587167 14843 tablet_service.cc:1514] Processing DeleteTablet for tablet 087e45ac11fa43f7a756c722033d184b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:09 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:33702
I20260927 10:45:09.587299 14976 tablet_service.cc:1514] Processing DeleteTablet for tablet 087e45ac11fa43f7a756c722033d184b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:09 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:53758
I20260927 10:45:09.587332 15109 tablet_service.cc:1514] Processing DeleteTablet for tablet 087e45ac11fa43f7a756c722033d184b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:09 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:36938
I20260927 10:45:09.587867 15214 tablet_replica.cc:330] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f: stopping tablet replica
I20260927 10:45:09.587942 15215 tablet_replica.cc:330] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: stopping tablet replica
I20260927 10:45:09.588249 15214 raft_consensus.cc:2229] T 087e45ac11fa43f7a756c722033d184b P 41a538e5b7354916ad56596264facb1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:09.588299 15215 raft_consensus.cc:2229] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:09.588573 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14781
I20260927 10:45:09.589273 15215 raft_consensus.cc:2258] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:09.589677 15216 tablet_replica.cc:330] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: stopping tablet replica
I20260927 10:45:09.589946 15216 raft_consensus.cc:2229] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:09.590219 15216 raft_consensus.cc:2258] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:09.591580 15216 ts_tablet_manager.cc:1905] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:09.593258 15215 ts_tablet_manager.cc:1905] T 087e45ac11fa43f7a756c722033d184b P e81dc95df30f4d83a6eeb70fc173d275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:09.594376 15216 ts_tablet_manager.cc:1918] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:09.594480 15216 log.cc:1196] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/087e45ac11fa43f7a756c722033d184b
I20260927 10:45:09.594808 15216 ts_tablet_manager.cc:1939] T 087e45ac11fa43f7a756c722033d184b P e95ec4fcb56c45d79b2635a5fa28d5f7: Deleting consensus metadata
I20260927 10:45:09.595976 14706 catalog_manager.cc:4841] TS e95ec4fcb56c45d79b2635a5fa28d5f7 (127.10.96.195:41587): tablet 087e45ac11fa43f7a756c722033d184b (table test-table [id=f0ef90864fbd42718d33e7422d35aa21]) successfully deleted
W20260927 10:45:09.597368 14703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:41029 (error 108)
I20260927 10:45:09.598130 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14914
I20260927 10:45:09.598332 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.598039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:41029 (local address 127.0.0.1:33706)
0927 10:45:09.598163 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.598239 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:41029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:45:09.599148 14703 catalog_manager.cc:4584] TS 41a538e5b7354916ad56596264facb1f (127.10.96.193:41029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 087e45ac11fa43f7a756c722033d184b: Network error: Client connection negotiation failed: client connection to 127.10.96.193:41029: connect: Connection refused (error 111)
I20260927 10:45:09.604508 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15047
I20260927 10:45:09.605935 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.604757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39965 (local address 127.0.0.1:53772)
0927 10:45:09.605778 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:09.605843 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:39965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":914}
I20260927 10:45:09.609871 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 14688
2026-09-27T10:45:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2339 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T10:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:09Z Disabled control of system clock
I20260927 10:45:09.639606 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36705
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:36705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:45:09.747377 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:09.747635 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:09.747673 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:09.752190 15221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:09.752249 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:09.752265 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:09.752302 15221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:09.752336 15221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:09.757176 15221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:36705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:09.758279 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:09.759639 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:09.768015 15229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.768023 15227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.767997 15226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:09.768075 15221 server_base.cc:1034] running on GCE node
I20260927 10:45:09.768965 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:09.770213 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:09.771399 15221 hybrid_clock.cc:648] HybridClock initialized: now 1790505909771379 us; error 34 us; skew 500 ppm
I20260927 10:45:09.773207 15221 webserver.cc:460] Webserver started at http://127.10.96.254:36277/ using document root <none> and password file <none>
I20260927 10:45:09.773829 15221 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:09.773890 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:09.774101 15221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:09.776086 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "6647978e5569400f87c0fea4afafb323"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "6130738192c3d907ca78817cfeb2bc1b"
server_key_iv: "3f22eb7e1c734d0a5616e0ec16d85922"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.776630 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "6647978e5569400f87c0fea4afafb323"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "6130738192c3d907ca78817cfeb2bc1b"
server_key_iv: "3f22eb7e1c734d0a5616e0ec16d85922"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.780249 15221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:45:09.782688 15235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:09.783869 15221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:09.784016 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "6647978e5569400f87c0fea4afafb323"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "6130738192c3d907ca78817cfeb2bc1b"
server_key_iv: "3f22eb7e1c734d0a5616e0ec16d85922"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.784129 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:09.788641 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:09.789212 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:09.789386 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:09.795723 15221 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:36705
I20260927 10:45:09.795729 15287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:36705 every 8 connection(s)
I20260927 10:45:09.796608 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:09.799712 15288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:09.804900 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15221
I20260927 10:45:09.805022 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:09.805114 15288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323: Bootstrap starting.
I20260927 10:45:09.805323 10627 external_mini_cluster.cc:1428] Setting key 4b1a59abb8e9f32de052ab56d4989631
I20260927 10:45:09.807619 15288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:09.808331 15288 log.cc:824] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:09.810483 15288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323: No bootstrap required, opened a new log
I20260927 10:45:09.810976 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:09.806649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.10.96.254:36705)
0927 10:45:09.807289 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:09.807304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:09.807337 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:45:09.808001 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:09.808020 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:09.808366 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:09.808752 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:09.809546 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:09.809552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:09.809566 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:09.809579 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:09.809641 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:09.809672 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:09.809780 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:09.809877 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 10:45:09.810110 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260927 10:45:09.814364 15288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6647978e5569400f87c0fea4afafb323" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36705 } }
I20260927 10:45:09.814564 15288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:09.814596 15288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6647978e5569400f87c0fea4afafb323, State: Initialized, Role: FOLLOWER
I20260927 10:45:09.814980 15288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6647978e5569400f87c0fea4afafb323" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36705 } }
I20260927 10:45:09.815094 15288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:09.815137 15288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:09.815212 15288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:09.815981 15288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6647978e5569400f87c0fea4afafb323" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36705 } }
I20260927 10:45:09.816267 15288 leader_election.cc:302] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6647978e5569400f87c0fea4afafb323; no voters:
I20260927 10:45:09.816507 15288 leader_election.cc:288] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:09.816632 15293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:09.816869 15293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [term 1 LEADER]: Becoming Leader. State: Replica: 6647978e5569400f87c0fea4afafb323, State: Running, Role: LEADER
I20260927 10:45:09.817222 15293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6647978e5569400f87c0fea4afafb323" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36705 } }
I20260927 10:45:09.817786 15288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:09.818882 15294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6647978e5569400f87c0fea4afafb323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6647978e5569400f87c0fea4afafb323" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36705 } } }
I20260927 10:45:09.819006 15294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:09.819350 15295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6647978e5569400f87c0fea4afafb323. Latest consensus state: current_term: 1 leader_uuid: "6647978e5569400f87c0fea4afafb323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6647978e5569400f87c0fea4afafb323" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36705 } } }
I20260927 10:45:09.819479 15295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:09.819831 15302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:09.822525 15302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:09.827630 15302 catalog_manager.cc:1347] Generated new cluster ID: 9367005dfefc448ba335d654aa8dbbef
I20260927 10:45:09.827703 15302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:09.836040 15302 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:09.836323 15302 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:09.846241 15302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323: Generated new TSK 0
I20260927 10:45:09.846915 15302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:09.854250 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36705
--builtin_ntp_servers=127.10.96.212:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 10:45:09.965653 15312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:09.965878 15312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:09.965947 15312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:09.969892 15312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:09.969940 15312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:09.970021 15312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:09.974695 15312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:09.975716 15312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:09.977056 15312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:09.986248 15317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.986238 15318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:09.986236 15320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:09.986584 15312 server_base.cc:1034] running on GCE node
I20260927 10:45:09.987015 15312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:09.987639 15312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:09.988823 15312 hybrid_clock.cc:648] HybridClock initialized: now 1790505909988791 us; error 54 us; skew 500 ppm
I20260927 10:45:09.990903 15312 webserver.cc:460] Webserver started at http://127.10.96.193:37453/ using document root <none> and password file <none>
I20260927 10:45:09.991482 15312 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:09.991571 15312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:09.991869 15312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:09.994513 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "05626bb69c7d462daf42eb6773c2d905"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "a02e00fefe66cb23b356cf128a8e8ad8"
server_key_iv: "ce65fef33b7b198a655dc0d677d21828"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.995054 15312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "05626bb69c7d462daf42eb6773c2d905"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "a02e00fefe66cb23b356cf128a8e8ad8"
server_key_iv: "ce65fef33b7b198a655dc0d677d21828"
server_key_version: "encryptionkey@0"
I20260927 10:45:09.998538 15312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:45:10.000977 15326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.002213 15312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:10.002367 15312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "05626bb69c7d462daf42eb6773c2d905"
format_stamp: "Formatted at 2026-09-27 10:45:09 on dist-test-slave-wmg0"
server_key: "a02e00fefe66cb23b356cf128a8e8ad8"
server_key_iv: "ce65fef33b7b198a655dc0d677d21828"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.002475 15312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:10.007133 15312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:10.007697 15312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:10.007869 15312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:10.008376 15312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:10.009198 15312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:10.009274 15312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.009359 15312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:10.009409 15312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.017693 15439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:33061 every 8 connection(s)
I20260927 10:45:10.017709 15312 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:33061
I20260927 10:45:10.018684 15312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:10.020038 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15312
I20260927 10:45:10.020157 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:10.020426 10627 external_mini_cluster.cc:1428] Setting key 8a042ad4d44ce109997ce538a0a4a0f2
I20260927 10:45:10.022310 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36705
--builtin_ntp_servers=127.10.96.212:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:45:10.025065 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.020829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:56897 (local address 127.10.96.254:36705)
0927 10:45:10.020962 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.020966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.021985 (+ 1019us) server_negotiation.cc:407] Connection header received
0927 10:45:10.023327 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.023331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.023371 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.023479 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.024648 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.024651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.024653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.024663 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.024684 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.024691 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.024777 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.024930 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.024966 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:45:10.026216 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.021074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36705 (local address 127.10.96.193:56897)
0927 10:45:10.021830 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.021878 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.023120 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.023636 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.023643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.023743 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.024351 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.024355 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.024409 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.024495 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.024782 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.024793 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.024801 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.024969 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.025193 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260927 10:45:10.027515 15440 heartbeater.cc:347] Connected to a master server at 127.10.96.254:36705
I20260927 10:45:10.027814 15440 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:10.028201 15440 heartbeater.cc:511] Master 127.10.96.254:36705 requested a full tablet report, sending...
I20260927 10:45:10.029883 15252 ts_manager.cc:194] Registered new tserver with Master: 05626bb69c7d462daf42eb6773c2d905 (127.10.96.193:33061)
W20260927 10:45:10.133678 15443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:10.133945 15443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:10.133984 15443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:10.137913 15443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:10.137965 15443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:10.138019 15443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:10.142653 15443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:10.143664 15443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:10.145001 15443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:10.154246 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:10.154384 15443 server_base.cc:1034] running on GCE node
W20260927 10:45:10.154210 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:10.154240 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:10.154851 15443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:10.155450 15443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:10.156635 15443 hybrid_clock.cc:648] HybridClock initialized: now 1790505910156615 us; error 33 us; skew 500 ppm
I20260927 10:45:10.158444 15443 webserver.cc:460] Webserver started at http://127.10.96.194:43065/ using document root <none> and password file <none>
I20260927 10:45:10.159010 15443 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:10.159096 15443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:10.159327 15443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:10.161273 15443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "7950ac5a867d7238ee751fb2eb9abb56"
server_key_iv: "2687286e5b74f7cb38f3cc5548912f9f"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.161902 15443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "7950ac5a867d7238ee751fb2eb9abb56"
server_key_iv: "2687286e5b74f7cb38f3cc5548912f9f"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.165450 15443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:10.167892 15457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.169186 15443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:10.169338 15443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "7950ac5a867d7238ee751fb2eb9abb56"
server_key_iv: "2687286e5b74f7cb38f3cc5548912f9f"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.169449 15443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:10.173846 15443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:10.174415 15443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:10.174587 15443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:10.175082 15443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:10.175928 15443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:10.176005 15443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.176090 15443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:10.176139 15443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.184610 15443 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:38467
I20260927 10:45:10.184636 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:38467 every 8 connection(s)
I20260927 10:45:10.185438 15443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:10.189031 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15443
I20260927 10:45:10.189165 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:10.189435 10627 external_mini_cluster.cc:1428] Setting key 537a8670ac575812c45f3598c1b0917c
I20260927 10:45:10.191233 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36705
--builtin_ntp_servers=127.10.96.212:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 10:45:10.191576 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.187192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:40137 (local address 127.10.96.254:36705)
0927 10:45:10.187365 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.187369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.188434 (+ 1065us) server_negotiation.cc:407] Connection header received
0927 10:45:10.189861 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.189865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.189909 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.190009 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.191094 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.191098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.191101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.191112 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.191133 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.191142 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.191246 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.191427 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.191472 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 10:45:10.192806 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.187611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36705 (local address 127.10.96.194:40137)
0927 10:45:10.188289 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.188328 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.189651 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.190161 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.190168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.190255 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.190847 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.190849 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.190885 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.190948 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.191244 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.191254 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.191261 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.191777 (+ 516us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.191954 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260927 10:45:10.193842 15571 heartbeater.cc:347] Connected to a master server at 127.10.96.254:36705
I20260927 10:45:10.194058 15571 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:10.194468 15571 heartbeater.cc:511] Master 127.10.96.254:36705 requested a full tablet report, sending...
I20260927 10:45:10.195489 15252 ts_manager.cc:194] Registered new tserver with Master: 2eb2cfc56a7f445f86e5da540d8ff0c0 (127.10.96.194:38467)
W20260927 10:45:10.300288 15574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:10.300562 15574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:10.300640 15574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:10.304719 15574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:10.304774 15574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:10.304862 15574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:10.309937 15574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:10.310998 15574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:10.312350 15574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:10.321759 15580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:10.321771 15579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:10.321768 15582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:10.321988 15574 server_base.cc:1034] running on GCE node
I20260927 10:45:10.322438 15574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:10.323000 15574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:10.324159 15574 hybrid_clock.cc:648] HybridClock initialized: now 1790505910324136 us; error 32 us; skew 500 ppm
I20260927 10:45:10.326000 15574 webserver.cc:460] Webserver started at http://127.10.96.195:35731/ using document root <none> and password file <none>
I20260927 10:45:10.326521 15574 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:10.326581 15574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:10.326742 15574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:10.328738 15574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "78b3a771eb8543d1a549ef94a9988eaf"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "b9bc510a04f3478e3219f62da66b6752"
server_key_iv: "aca554590b6120c9c0f58faf74c66852"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.329303 15574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "78b3a771eb8543d1a549ef94a9988eaf"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "b9bc510a04f3478e3219f62da66b6752"
server_key_iv: "aca554590b6120c9c0f58faf74c66852"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.333016 15574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 10:45:10.335520 15588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.336757 15574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:10.336895 15574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "78b3a771eb8543d1a549ef94a9988eaf"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "b9bc510a04f3478e3219f62da66b6752"
server_key_iv: "aca554590b6120c9c0f58faf74c66852"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.337006 15574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:10.341420 15574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:10.342056 15574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:10.342235 15574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:10.342753 15574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:10.343623 15574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:10.343693 15574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.343756 15574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:10.343818 15574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.352792 15574 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:39921
I20260927 10:45:10.352814 15701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:39921 every 8 connection(s)
I20260927 10:45:10.353710 15574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:10.357409 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15574
I20260927 10:45:10.357581 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:10.357851 10627 external_mini_cluster.cc:1428] Setting key 93967b202ed96da41833dc078c414d78
I20260927 10:45:10.359947 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.355462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:47887 (local address 127.10.96.254:36705)
0927 10:45:10.355585 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.355588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.356631 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 10:45:10.358050 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.358054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.358097 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.358175 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.359414 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.359416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.359419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.359428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.359447 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.359455 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.359521 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.359792 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.359835 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:45:10.360760 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.355867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36705 (local address 127.10.96.195:47887)
0927 10:45:10.356486 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.356525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.357822 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.358332 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.358339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.358454 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.359127 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.359129 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.359183 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.359265 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.359539 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.359547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.359553 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.359729 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.360003 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260927 10:45:10.361814 15702 heartbeater.cc:347] Connected to a master server at 127.10.96.254:36705
I20260927 10:45:10.362006 15702 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:10.362370 15702 heartbeater.cc:511] Master 127.10.96.254:36705 requested a full tablet report, sending...
I20260927 10:45:10.363293 15252 ts_manager.cc:194] Registered new tserver with Master: 78b3a771eb8543d1a549ef94a9988eaf (127.10.96.195:39921)
I20260927 10:45:10.371595 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:45:10.375020 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.373490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.10.96.254:36705)
0927 10:45:10.373618 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.373624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.373847 (+ 223us) server_negotiation.cc:407] Connection header received
0927 10:45:10.373930 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.373933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.373966 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.374050 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.374644 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.374647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.374649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.374659 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.374684 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.374692 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.374760 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.374828 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.374860 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:45:10.377590 15252 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:10.380455 15252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:10.392109 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.388904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.10.96.193:33061)
0927 10:45:10.389206 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.389218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.389516 (+ 298us) server_negotiation.cc:407] Connection header received
0927 10:45:10.389701 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.389708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.389908 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.390025 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.391558 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.391564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.391576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.391588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.391631 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.391652 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.391766 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.391874 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.391942 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
I20260927 10:45:10.392105 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.388653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:38467 (local address 127.0.0.1:56588)
0927 10:45:10.389357 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.389404 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.389545 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.390375 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.390381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.390457 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.391045 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.391047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.391066 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.391130 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.391787 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.391791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.391794 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.391852 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.391953 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
I20260927 10:45:10.392105 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.389739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:39921 (local address 127.0.0.1:39026)
0927 10:45:10.390292 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.390308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.390372 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.390976 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.390980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.390999 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.391388 (+ 389us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.391390 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.391398 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.391425 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.391787 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.391791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.391794 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.391852 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.391942 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":152,"threads_started":1}
I20260927 10:45:10.392446 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.388764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.10.96.194:38467)
0927 10:45:10.389907 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.389920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.389943 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:45:10.389996 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.390000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.390166 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.390731 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.391595 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.391601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.391611 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.391621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.391656 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.391677 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.392162 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.392230 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.392313 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":114,"threads_started":1}
I20260927 10:45:10.392097 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.388733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:33061 (local address 127.0.0.1:47618)
0927 10:45:10.389368 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.389405 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.389575 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.390723 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.390727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.390745 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.391209 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.391212 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.391223 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.391259 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.391744 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.391753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.391759 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.391862 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.391923 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":99,"threads_started":1}
I20260927 10:45:10.393534 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.390201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.10.96.195:39921)
0927 10:45:10.390597 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.390610 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.390635 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:45:10.390688 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.390712 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.390862 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.390991 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.391595 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.391601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.391611 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.391621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.391656 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.391677 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.393266 (+ 1589us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.393317 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.393368 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
I20260927 10:45:10.395406 15374 tablet_service.cc:1467] Processing CreateTablet for tablet cf193122b53b4e599944327cf4bbb349 (DEFAULT_TABLE table=test-table [id=54b91f2ba3d94f449370eff49d7192cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:10.395406 15505 tablet_service.cc:1467] Processing CreateTablet for tablet cf193122b53b4e599944327cf4bbb349 (DEFAULT_TABLE table=test-table [id=54b91f2ba3d94f449370eff49d7192cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:10.395728 15636 tablet_service.cc:1467] Processing CreateTablet for tablet cf193122b53b4e599944327cf4bbb349 (DEFAULT_TABLE table=test-table [id=54b91f2ba3d94f449370eff49d7192cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:10.396514 15505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf193122b53b4e599944327cf4bbb349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:10.396514 15374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf193122b53b4e599944327cf4bbb349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:10.396704 15636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf193122b53b4e599944327cf4bbb349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:10.402330 15719 tablet_bootstrap.cc:492] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Bootstrap starting.
I20260927 10:45:10.404065 15720 tablet_bootstrap.cc:492] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905: Bootstrap starting.
I20260927 10:45:10.404812 15719 tablet_bootstrap.cc:654] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:10.405655 15721 tablet_bootstrap.cc:492] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Bootstrap starting.
I20260927 10:45:10.405676 15719 log.cc:824] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:10.406188 15720 tablet_bootstrap.cc:654] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:10.406978 15720 log.cc:824] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:10.407699 15721 tablet_bootstrap.cc:654] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:10.407846 15719 tablet_bootstrap.cc:492] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: No bootstrap required, opened a new log
I20260927 10:45:10.408025 15719 ts_tablet_manager.cc:1397] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:45:10.408466 15721 log.cc:824] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:10.408823 15720 tablet_bootstrap.cc:492] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905: No bootstrap required, opened a new log
I20260927 10:45:10.408998 15720 ts_tablet_manager.cc:1397] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:45:10.410262 15721 tablet_bootstrap.cc:492] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: No bootstrap required, opened a new log
I20260927 10:45:10.410449 15721 ts_tablet_manager.cc:1397] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:45:10.412452 15719 raft_consensus.cc:348] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.412720 15719 raft_consensus.cc:374] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:10.412772 15719 raft_consensus.cc:729] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b3a771eb8543d1a549ef94a9988eaf, State: Initialized, Role: FOLLOWER
I20260927 10:45:10.413198 15719 consensus_queue.cc:260] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.413344 15720 raft_consensus.cc:348] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.413599 15720 raft_consensus.cc:374] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:10.413652 15720 raft_consensus.cc:729] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05626bb69c7d462daf42eb6773c2d905, State: Initialized, Role: FOLLOWER
I20260927 10:45:10.413760 15721 raft_consensus.cc:348] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.413964 15702 heartbeater.cc:503] Master 127.10.96.254:36705 was elected leader, sending a full tablet report...
I20260927 10:45:10.413980 15721 raft_consensus.cc:374] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:10.414022 15721 raft_consensus.cc:729] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb2cfc56a7f445f86e5da540d8ff0c0, State: Initialized, Role: FOLLOWER
I20260927 10:45:10.414072 15720 consensus_queue.cc:260] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.414203 15719 ts_tablet_manager.cc:1428] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 10:45:10.414389 15721 consensus_queue.cc:260] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.414887 15440 heartbeater.cc:503] Master 127.10.96.254:36705 was elected leader, sending a full tablet report...
I20260927 10:45:10.414994 15720 ts_tablet_manager.cc:1428] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:45:10.415232 15721 ts_tablet_manager.cc:1428] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:45:10.415849 15571 heartbeater.cc:503] Master 127.10.96.254:36705 was elected leader, sending a full tablet report...
W20260927 10:45:10.436383 15572 tablet.cc:2369] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:45:10.459565 15726 raft_consensus.cc:482] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:10.459806 15726 raft_consensus.cc:504] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.460947 15726 leader_election.cc:288] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eb2cfc56a7f445f86e5da540d8ff0c0 (127.10.96.194:38467), 78b3a771eb8543d1a549ef94a9988eaf (127.10.96.195:39921)
I20260927 10:45:10.462795 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.461141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:38467 (local address 127.10.96.193:55719)
0927 10:45:10.461281 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.461299 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.461382 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.461665 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.461667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.461679 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.462275 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.462277 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.462286 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.462309 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.462552 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.462555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.462558 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.462639 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.462675 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 10:45:10.462803 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.461264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55719 (local address 127.10.96.194:38467)
0927 10:45:10.461404 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.461408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.461426 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:45:10.461500 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.461504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.461545 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.461634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.462431 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.462434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.462437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.462452 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.462468 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.462475 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.462567 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.462651 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.462687 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:45:10.462980 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.461426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:40469 (local address 127.10.96.195:39921)
0927 10:45:10.461573 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.461576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.461732 (+ 156us) server_negotiation.cc:407] Connection header received
0927 10:45:10.461810 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.461814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.461855 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.461931 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.462604 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.462607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.462609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.462621 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.462647 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.462655 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.462750 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.462822 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.462859 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:45:10.463008 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.461212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:39921 (local address 127.10.96.193:40469)
0927 10:45:10.461629 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.461642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.461732 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.461968 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.461972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.461987 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:10.462445 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:10.462447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:10.462455 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:10.462478 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:10.462736 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:10.462739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:10.462742 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:45:10.462823 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:45:10.462859 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":148,"threads_started":1}
I20260927 10:45:10.463515 15525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf193122b53b4e599944327cf4bbb349" candidate_uuid: "05626bb69c7d462daf42eb6773c2d905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" is_pre_election: true
I20260927 10:45:10.463626 15656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf193122b53b4e599944327cf4bbb349" candidate_uuid: "05626bb69c7d462daf42eb6773c2d905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b3a771eb8543d1a549ef94a9988eaf" is_pre_election: true
I20260927 10:45:10.463837 15525 raft_consensus.cc:2454] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05626bb69c7d462daf42eb6773c2d905 in term 0.
I20260927 10:45:10.463910 15656 raft_consensus.cc:2454] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05626bb69c7d462daf42eb6773c2d905 in term 0.
I20260927 10:45:10.464309 15329 leader_election.cc:302] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05626bb69c7d462daf42eb6773c2d905, 2eb2cfc56a7f445f86e5da540d8ff0c0; no voters:
I20260927 10:45:10.464581 15726 raft_consensus.cc:2781] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:10.464659 15726 raft_consensus.cc:482] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:10.464701 15726 raft_consensus.cc:3037] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:10.465729 15726 raft_consensus.cc:504] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.466039 15726 leader_election.cc:288] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [CANDIDATE]: Term 1 election: Requested vote from peers 2eb2cfc56a7f445f86e5da540d8ff0c0 (127.10.96.194:38467), 78b3a771eb8543d1a549ef94a9988eaf (127.10.96.195:39921)
I20260927 10:45:10.466321 15525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf193122b53b4e599944327cf4bbb349" candidate_uuid: "05626bb69c7d462daf42eb6773c2d905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0"
I20260927 10:45:10.466462 15525 raft_consensus.cc:3037] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:10.466463 15656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf193122b53b4e599944327cf4bbb349" candidate_uuid: "05626bb69c7d462daf42eb6773c2d905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78b3a771eb8543d1a549ef94a9988eaf"
I20260927 10:45:10.466598 15656 raft_consensus.cc:3037] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:10.467482 15525 raft_consensus.cc:2454] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05626bb69c7d462daf42eb6773c2d905 in term 1.
I20260927 10:45:10.467654 15656 raft_consensus.cc:2454] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05626bb69c7d462daf42eb6773c2d905 in term 1.
I20260927 10:45:10.467723 15329 leader_election.cc:302] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05626bb69c7d462daf42eb6773c2d905, 2eb2cfc56a7f445f86e5da540d8ff0c0; no voters:
I20260927 10:45:10.467876 15726 raft_consensus.cc:2781] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:10.468112 15726 raft_consensus.cc:686] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [term 1 LEADER]: Becoming Leader. State: Replica: 05626bb69c7d462daf42eb6773c2d905, State: Running, Role: LEADER
I20260927 10:45:10.468425 15726 consensus_queue.cc:237] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } }
I20260927 10:45:10.471462 15249 catalog_manager.cc:5495] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05626bb69c7d462daf42eb6773c2d905 (127.10.96.193). New cstate: current_term: 1 leader_uuid: "05626bb69c7d462daf42eb6773c2d905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05626bb69c7d462daf42eb6773c2d905" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 33061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:10.488991 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.487624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47620 (local address 127.10.96.193:33061)
0927 10:45:10.487793 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.487796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.487808 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:10.487916 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.487919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.487971 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.488038 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.488643 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.488646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.488648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.488659 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.488677 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.488684 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.488770 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.488850 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.488898 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:45:10.494354 15656 raft_consensus.cc:1261] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 1 FOLLOWER]: Refusing update from remote peer 05626bb69c7d462daf42eb6773c2d905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:10.494354 15525 raft_consensus.cc:1261] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 1 FOLLOWER]: Refusing update from remote peer 05626bb69c7d462daf42eb6773c2d905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:10.495057 15729 consensus_queue.cc:1035] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb2cfc56a7f445f86e5da540d8ff0c0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:10.495225 15726 consensus_queue.cc:1035] T cf193122b53b4e599944327cf4bbb349 P 05626bb69c7d462daf42eb6773c2d905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78b3a771eb8543d1a549ef94a9988eaf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:10.502575 15732 mvcc.cc:205] Tried to move back new op lower bound from 7333912209378152448 to 7333912209280888832. Current Snapshot: MvccSnapshot[applied={T|T < 7333912209378152448}]
I20260927 10:45:10.504347 15734 mvcc.cc:205] Tried to move back new op lower bound from 7333912209378152448 to 7333912209280888832. Current Snapshot: MvccSnapshot[applied={T|T < 7333912209378152448}]
I20260927 10:45:10.510177 15249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:10.510344 15249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:10.512650 15249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6647978e5569400f87c0fea4afafb323: Sending DeleteTablet for 3 replicas of tablet cf193122b53b4e599944327cf4bbb349
I20260927 10:45:10.513355 15505 tablet_service.cc:1514] Processing DeleteTablet for tablet cf193122b53b4e599944327cf4bbb349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:10 UTC) from {username='slave'} at 127.0.0.1:56588
I20260927 10:45:10.513391 15374 tablet_service.cc:1514] Processing DeleteTablet for tablet cf193122b53b4e599944327cf4bbb349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:10 UTC) from {username='slave'} at 127.0.0.1:47618
I20260927 10:45:10.513827 15741 tablet_replica.cc:330] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: stopping tablet replica
I20260927 10:45:10.513787 15636 tablet_service.cc:1514] Processing DeleteTablet for tablet cf193122b53b4e599944327cf4bbb349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:10 UTC) from {username='slave'} at 127.0.0.1:39026
I20260927 10:45:10.514118 15741 raft_consensus.cc:2229] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:10.514191 15743 tablet_replica.cc:330] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: stopping tablet replica
I20260927 10:45:10.514410 15741 raft_consensus.cc:2258] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:10.514472 15743 raft_consensus.cc:2229] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:10.514729 15743 raft_consensus.cc:2258] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:10.514868 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15312
I20260927 10:45:10.516515 15743 ts_tablet_manager.cc:1905] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:10.516541 15741 ts_tablet_manager.cc:1905] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:10.518867 15743 ts_tablet_manager.cc:1918] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:10.519013 15743 log.cc:1196] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/cf193122b53b4e599944327cf4bbb349
I20260927 10:45:10.519383 15743 ts_tablet_manager.cc:1939] T cf193122b53b4e599944327cf4bbb349 P 78b3a771eb8543d1a549ef94a9988eaf: Deleting consensus metadata
I20260927 10:45:10.520035 15741 ts_tablet_manager.cc:1918] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:10.520129 15741 log.cc:1196] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/cf193122b53b4e599944327cf4bbb349
I20260927 10:45:10.520442 15741 ts_tablet_manager.cc:1939] T cf193122b53b4e599944327cf4bbb349 P 2eb2cfc56a7f445f86e5da540d8ff0c0: Deleting consensus metadata
I20260927 10:45:10.520522 15236 catalog_manager.cc:4841] TS 78b3a771eb8543d1a549ef94a9988eaf (127.10.96.195:39921): tablet cf193122b53b4e599944327cf4bbb349 (table test-table [id=54b91f2ba3d94f449370eff49d7192cb]) successfully deleted
I20260927 10:45:10.521622 15238 catalog_manager.cc:4841] TS 2eb2cfc56a7f445f86e5da540d8ff0c0 (127.10.96.194:38467): tablet cf193122b53b4e599944327cf4bbb349 (table test-table [id=54b91f2ba3d94f449370eff49d7192cb]) successfully deleted
I20260927 10:45:10.523986 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15443
W20260927 10:45:10.523967 15237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:33061 (error 108)
I20260927 10:45:10.525424 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.525081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:33061 (local address 127.0.0.1:47628)
0927 10:45:10.525229 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.525318 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:33061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 10:45:10.526458 15237 catalog_manager.cc:4584] TS 05626bb69c7d462daf42eb6773c2d905 (127.10.96.193:33061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf193122b53b4e599944327cf4bbb349: Network error: Client connection negotiation failed: client connection to 127.10.96.193:33061: connect: Connection refused (error 111)
I20260927 10:45:10.530371 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15574
I20260927 10:45:10.535475 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15221
2026-09-27T10:45:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (924 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 10:45:10.556509 10627 mini_oidc.cc:152] Starting JWKS server
I20260927 10:45:10.558146 10627 webserver.cc:460] Webserver started at https://127.0.0.1:35661/ using document root <none> and password file <none>
I20260927 10:45:10.558279 10627 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:45:10.558493 10627 webserver.cc:460] Webserver started at http://127.0.0.1:35697/ using document root <none> and password file <none>
2026-09-27T10:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:10Z Disabled control of system clock
I20260927 10:45:10.567167 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:38433
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:38433
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35661/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/testchainca.cert with env {}
W20260927 10:45:10.695906 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:10.696317 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:10.696411 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:10.702159 15750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:10.702230 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:10.702246 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:10.702265 15750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:10.702279 15750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:45:10.702296 15750 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:45:10.702309 15750 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35661/jwks.json
I20260927 10:45:10.707887 15750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:38433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:38433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35661/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:10.708976 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:10.710080 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:10.718806 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:10.718779 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:10.718835 15750 server_base.cc:1034] running on GCE node
W20260927 10:45:10.718782 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:10.719558 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:10.720702 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:10.721891 15750 hybrid_clock.cc:648] HybridClock initialized: now 1790505910721861 us; error 46 us; skew 500 ppm
I20260927 10:45:10.723783 15750 webserver.cc:460] Webserver started at http://127.10.96.254:45261/ using document root <none> and password file <none>
I20260927 10:45:10.724380 15750 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:10.724470 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:10.724695 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:10.726783 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "913dd385da2642fa84477ebe1bdc8315"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "49fd009de73341e38e8e677762d1f003"
server_key_iv: "6d3d0a46f2f55981491d05bf1a983af3"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.727348 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "913dd385da2642fa84477ebe1bdc8315"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "49fd009de73341e38e8e677762d1f003"
server_key_iv: "6d3d0a46f2f55981491d05bf1a983af3"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.731019 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:10.733683 15764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:10.735064 15750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:10.735224 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "913dd385da2642fa84477ebe1bdc8315"
format_stamp: "Formatted at 2026-09-27 10:45:10 on dist-test-slave-wmg0"
server_key: "49fd009de73341e38e8e677762d1f003"
server_key_iv: "6d3d0a46f2f55981491d05bf1a983af3"
server_key_version: "encryptionkey@0"
I20260927 10:45:10.735339 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:10.772954 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:10.773824 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:10.774025 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:10.780853 15816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:38433 every 8 connection(s)
I20260927 10:45:10.780834 15750 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:38433
I20260927 10:45:10.781985 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:10.783120 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15750
I20260927 10:45:10.783318 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:10.783592 10627 external_mini_cluster.cc:1428] Setting key 63d72ab7cd196bc9a4a44d5d48fbda29
I20260927 10:45:10.785631 15817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:10.791335 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315: Bootstrap starting.
I20260927 10:45:10.792912 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.784968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.10.96.254:38433)
0927 10:45:10.785733 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.785762 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.785800 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:45:10.786450 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.786467 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.786850 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.787190 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:10.787631 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:10.788605 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:10.789608 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:10.790084 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:10.790254 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:10.791165 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:10.791170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:10.791183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:10.791611 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:10.791666 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:10.791686 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:10.791791 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:10.791947 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:45:10.792198 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":136,"threads_started":1}
I20260927 10:45:10.793978 15817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:10.794772 15817 log.cc:824] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:10.795769 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:45:10.797138 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315: No bootstrap required, opened a new log
I20260927 10:45:10.800925 15817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913dd385da2642fa84477ebe1bdc8315" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 38433 } }
I20260927 10:45:10.801204 15817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:10.801245 15817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913dd385da2642fa84477ebe1bdc8315, State: Initialized, Role: FOLLOWER
I20260927 10:45:10.801738 15817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913dd385da2642fa84477ebe1bdc8315" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 38433 } }
I20260927 10:45:10.801864 15817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:10.801908 15817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:10.802014 15817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:10.803234 15817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913dd385da2642fa84477ebe1bdc8315" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 38433 } }
I20260927 10:45:10.803524 15817 leader_election.cc:302] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913dd385da2642fa84477ebe1bdc8315; no voters:
I20260927 10:45:10.803778 15817 leader_election.cc:288] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:10.803911 15822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:10.804162 15822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [term 1 LEADER]: Becoming Leader. State: Replica: 913dd385da2642fa84477ebe1bdc8315, State: Running, Role: LEADER
I20260927 10:45:10.804471 15822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913dd385da2642fa84477ebe1bdc8315" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 38433 } }
I20260927 10:45:10.804967 15817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:10.806274 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913dd385da2642fa84477ebe1bdc8315. Latest consensus state: current_term: 1 leader_uuid: "913dd385da2642fa84477ebe1bdc8315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913dd385da2642fa84477ebe1bdc8315" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 38433 } } }
I20260927 10:45:10.806430 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:10.806288 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913dd385da2642fa84477ebe1bdc8315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913dd385da2642fa84477ebe1bdc8315" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 38433 } } }
I20260927 10:45:10.806536 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:10.807013 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:10.810374 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:10.815501 15831 catalog_manager.cc:1347] Generated new cluster ID: b87c46f784ec41bfb152c5f74c8bac74
I20260927 10:45:10.815562 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:10.825937 15831 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:10.826848 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:10.831478 15831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 913dd385da2642fa84477ebe1bdc8315: Generated new TSK 0
I20260927 10:45:10.832160 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:45:10.870437 15847 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 10:45:10.870558 15847 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:45:10.871024 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:10.846304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.10.96.254:38433)
0927 10:45:10.846436 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:10.846441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:10.846670 (+ 229us) server_negotiation.cc:407] Connection header received
0927 10:45:10.846880 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:10.846883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:10.846932 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:10.847023 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:45:10.847355 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:10.847961 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:10.849238 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:10.849447 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:10.849611 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:10.850014 (+ 403us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:45:10.870610 (+ 20596us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:45:10.870884 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:45:10.871114 15820 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 10:45:10.872424 15846 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:10.846167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:38433 (local address 127.0.0.1:51148)
0927 10:45:10.846544 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:10.846560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:10.846694 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:10.847065 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:10.847068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:10.847076 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:45:10.847218 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:10.847225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:10.848089 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:10.848093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:10.849079 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:10.849089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:10.849640 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:10.849644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:10.849741 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:10.849795 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:45:10.870863 (+ 21068us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 10:45:10.871119 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.96.254:38433: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.96.254:38433: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35661/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 10:45:10.873343 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15750
2026-09-27T10:45:10Z chronyd exiting
I20260927 10:45:10.910686 10627 test_util.cc:175] -----------------------------------------------
I20260927 10:45:10.910799 10627 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790505884435945-10627-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (355 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 10:45:10.911813 10627 mini_oidc.cc:152] Starting JWKS server
I20260927 10:45:10.912664 10627 webserver.cc:460] Webserver started at https://127.0.0.1:33893/ using document root <none> and password file <none>
I20260927 10:45:10.912802 10627 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:45:10.913081 10627 webserver.cc:460] Webserver started at http://127.0.0.1:40757/ using document root <none> and password file <none>
2026-09-27T10:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:10Z Disabled control of system clock
I20260927 10:45:10.921811 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:45317
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:41697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:45317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/testchainca.cert with env {}
W20260927 10:45:11.028756 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:11.029004 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:11.029043 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:11.033218 15853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:11.033294 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:11.033310 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:11.033327 15853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:11.033342 15853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:45:11.033358 15853 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:45:11.033371 15853 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33893/jwks.json
I20260927 10:45:11.038635 15853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:45317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:45317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:11.039583 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:11.040512 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:11.048941 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:11.048950 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:11.048919 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:11.049113 15853 server_base.cc:1034] running on GCE node
I20260927 10:45:11.049806 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:11.050952 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:11.052126 15853 hybrid_clock.cc:648] HybridClock initialized: now 1790505911052111 us; error 27 us; skew 500 ppm
I20260927 10:45:11.054183 15853 webserver.cc:460] Webserver started at http://127.10.96.254:39707/ using document root <none> and password file <none>
I20260927 10:45:11.054755 15853 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:11.054818 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:11.055025 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:11.057077 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "922a40e5b0f94ea39278f3a713a1b9a2"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "7dec2e8dcd38db7268f6977c09689a85"
server_key_iv: "f4f9927f405ca54366bd4ff8a4ab5ebd"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.057785 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "922a40e5b0f94ea39278f3a713a1b9a2"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "7dec2e8dcd38db7268f6977c09689a85"
server_key_iv: "f4f9927f405ca54366bd4ff8a4ab5ebd"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.062285 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:45:11.064651 15867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:11.065917 15853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:11.066051 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "922a40e5b0f94ea39278f3a713a1b9a2"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "7dec2e8dcd38db7268f6977c09689a85"
server_key_iv: "f4f9927f405ca54366bd4ff8a4ab5ebd"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.066160 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:11.103268 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:11.104022 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:11.104235 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:11.111150 15853 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:45317
I20260927 10:45:11.111159 15919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:45317 every 8 connection(s)
I20260927 10:45:11.112130 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:11.115391 15920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:11.117710 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15853
I20260927 10:45:11.117836 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:11.118098 10627 external_mini_cluster.cc:1428] Setting key 57c604a7e712f15842dcbd562342b0af
I20260927 10:45:11.121035 15920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2: Bootstrap starting.
I20260927 10:45:11.123453 15920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:11.124315 15920 log.cc:824] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:11.126122 15920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2: No bootstrap required, opened a new log
I20260927 10:45:11.127544 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:11.119534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.10.96.254:45317)
0927 10:45:11.120136 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:11.120150 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:11.120191 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:45:11.120806 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:11.120822 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:11.121268 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:11.121672 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:11.122085 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.123037 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.124268 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.124782 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.124957 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.125873 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:11.125886 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:11.125899 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:11.126332 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:11.126385 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:11.126405 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:11.126511 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:11.126636 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 10:45:11.126882 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260927 10:45:11.129688 15920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45317 } }
I20260927 10:45:11.129894 15920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:11.129959 15920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 922a40e5b0f94ea39278f3a713a1b9a2, State: Initialized, Role: FOLLOWER
I20260927 10:45:11.130203 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:45:11.130434 15920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45317 } }
I20260927 10:45:11.130607 15920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:11.130677 15920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:11.130779 15920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:11.131724 15920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45317 } }
I20260927 10:45:11.132064 15920 leader_election.cc:302] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 922a40e5b0f94ea39278f3a713a1b9a2; no voters:
I20260927 10:45:11.132376 15920 leader_election.cc:288] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:11.132498 15925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:11.132723 15925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 922a40e5b0f94ea39278f3a713a1b9a2, State: Running, Role: LEADER
I20260927 10:45:11.133020 15925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45317 } }
I20260927 10:45:11.133649 15920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:11.134711 15927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45317 } } }
I20260927 10:45:11.134837 15927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:11.135198 15934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:11.135164 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 922a40e5b0f94ea39278f3a713a1b9a2. Latest consensus state: current_term: 1 leader_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "922a40e5b0f94ea39278f3a713a1b9a2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45317 } } }
I20260927 10:45:11.135361 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:11.138731 15934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:11.144680 15934 catalog_manager.cc:1347] Generated new cluster ID: 41aef54f9b134fd3bacacd5c13f6975e
I20260927 10:45:11.144749 15934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:11.165203 15934 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:11.166225 15934 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:11.172374 15934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 922a40e5b0f94ea39278f3a713a1b9a2: Generated new TSK 0
I20260927 10:45:11.173030 15934 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 10:45:11.198249 15950 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 10:45:11.198350 15950 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 10:45:11.199035 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:11.182759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47962 (local address 127.10.96.254:45317)
0927 10:45:11.182885 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:11.182889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:11.183036 (+ 147us) server_negotiation.cc:407] Connection header received
0927 10:45:11.183159 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:11.183162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:11.183224 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:11.183332 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:45:11.183528 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.184357 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.185187 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.185565 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.185664 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.185917 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:45:11.198626 (+ 12709us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:45:11.198897 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 10:45:11.199100 15923 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 10:45:11.199134 15949 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:11.182658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45317 (local address 127.0.0.1:47962)
0927 10:45:11.182958 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:11.182970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:11.183060 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:11.183333 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:11.183335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:11.183346 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:45:11.183437 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:11.183441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.184469 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.184472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:11.185083 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:11.185090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.185675 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.185678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:11.185740 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.185761 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:45:11.198828 (+ 13067us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 10:45:11.198965 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.96.254:45317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260927 10:45:11.199965 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15853
2026-09-27T10:45:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 10:45:11.238976 10627 mini_oidc.cc:152] Starting JWKS server
I20260927 10:45:11.239825 10627 webserver.cc:460] Webserver started at https://127.0.0.1:40565/ using document root <none> and password file <none>
I20260927 10:45:11.239933 10627 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 10:45:11.240199 10627 webserver.cc:460] Webserver started at http://127.0.0.1:33799/ using document root <none> and password file <none>
2026-09-27T10:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:11Z Disabled control of system clock
I20260927 10:45:11.248658 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:34713
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:34713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40565/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:45:11.359104 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:11.359382 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:11.359449 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:11.363415 15956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:11.363492 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:11.363519 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:11.363555 15956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:11.363586 15956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:45:11.363617 15956 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:45:11.363646 15956 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40565/jwks.json
I20260927 10:45:11.368346 15956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:34713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:34713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40565/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:11.369391 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:11.370740 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:11.378693 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:11.378693 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:11.378867 15956 server_base.cc:1034] running on GCE node
W20260927 10:45:11.378762 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:11.379446 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:11.380407 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:11.381525 15956 hybrid_clock.cc:648] HybridClock initialized: now 1790505911381512 us; error 26 us; skew 500 ppm
I20260927 10:45:11.383270 15956 webserver.cc:460] Webserver started at http://127.10.96.254:40167/ using document root <none> and password file <none>
I20260927 10:45:11.383841 15956 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:11.383900 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:11.384115 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:11.386149 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "e164719648bc42fba69b8acd182267ea"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "843813a497f86a1073a2f223f89cce35"
server_key_iv: "3ea839e58236f84e89d53d18a47188dd"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.386686 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "e164719648bc42fba69b8acd182267ea"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "843813a497f86a1073a2f223f89cce35"
server_key_iv: "3ea839e58236f84e89d53d18a47188dd"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.390300 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:11.392594 15970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:11.393810 15956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:11.393940 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "e164719648bc42fba69b8acd182267ea"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "843813a497f86a1073a2f223f89cce35"
server_key_iv: "3ea839e58236f84e89d53d18a47188dd"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.394047 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:11.416973 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:11.417686 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:11.417883 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:11.424371 15956 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:34713
I20260927 10:45:11.424383 16022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:34713 every 8 connection(s)
I20260927 10:45:11.425288 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:11.428301 16023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:11.433657 16023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea: Bootstrap starting.
I20260927 10:45:11.433938 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 15956
I20260927 10:45:11.434073 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:11.434347 10627 external_mini_cluster.cc:1428] Setting key ae12398ebdd2403a5988d809d2b6e41f
I20260927 10:45:11.436178 16023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:11.436895 16023 log.cc:824] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:11.438758 16023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea: No bootstrap required, opened a new log
I20260927 10:45:11.442375 16023 raft_consensus.cc:348] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e164719648bc42fba69b8acd182267ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34713 } }
I20260927 10:45:11.442641 16023 raft_consensus.cc:374] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:11.442709 16023 raft_consensus.cc:729] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e164719648bc42fba69b8acd182267ea, State: Initialized, Role: FOLLOWER
I20260927 10:45:11.443112 16023 consensus_queue.cc:260] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e164719648bc42fba69b8acd182267ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34713 } }
I20260927 10:45:11.443228 16023 raft_consensus.cc:388] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:11.443310 16023 raft_consensus.cc:482] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:11.443423 16023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:11.443602 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:11.435651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.10.96.254:34713)
0927 10:45:11.436223 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:11.436236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:11.436271 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:45:11.436908 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:11.436924 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:11.437412 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:11.437811 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:11.438220 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.439200 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.440062 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.440574 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.440747 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.441689 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:11.441694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:11.441705 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:11.442059 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:11.442110 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:11.442128 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:11.442232 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:11.442415 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 10:45:11.442686 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
I20260927 10:45:11.444301 16023 raft_consensus.cc:504] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e164719648bc42fba69b8acd182267ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34713 } }
I20260927 10:45:11.444629 16023 leader_election.cc:302] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e164719648bc42fba69b8acd182267ea; no voters:
I20260927 10:45:11.444944 16023 leader_election.cc:288] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:11.445005 16028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:11.445214 16028 raft_consensus.cc:686] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [term 1 LEADER]: Becoming Leader. State: Replica: e164719648bc42fba69b8acd182267ea, State: Running, Role: LEADER
I20260927 10:45:11.445567 16028 consensus_queue.cc:237] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e164719648bc42fba69b8acd182267ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34713 } }
I20260927 10:45:11.446165 16023 sys_catalog.cc:564] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:11.446357 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:45:11.447177 16029 sys_catalog.cc:455] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e164719648bc42fba69b8acd182267ea. Latest consensus state: current_term: 1 leader_uuid: "e164719648bc42fba69b8acd182267ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e164719648bc42fba69b8acd182267ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34713 } } }
I20260927 10:45:11.447290 16029 sys_catalog.cc:458] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:11.447566 16030 sys_catalog.cc:455] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e164719648bc42fba69b8acd182267ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e164719648bc42fba69b8acd182267ea" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34713 } } }
I20260927 10:45:11.447646 16037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:11.447654 16030 sys_catalog.cc:458] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:11.450824 16037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:11.455824 16037 catalog_manager.cc:1347] Generated new cluster ID: 7dc7b54fdff347ceb45591e351113e0f
I20260927 10:45:11.455895 16037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:11.466506 16037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:11.467340 16037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:11.479035 16037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e164719648bc42fba69b8acd182267ea: Generated new TSK 0
I20260927 10:45:11.479872 16037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:11.503094 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:11.498025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.10.96.254:34713)
0927 10:45:11.498141 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:11.498146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:11.498350 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:45:11.498526 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:11.498535 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:11.498585 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:11.498675 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:45:11.498932 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.499478 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.500688 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.500911 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.500988 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.501320 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:45:11.502725 (+ 1405us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:45:11.502949 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35632: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260927 10:45:11.503162 16026 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35632: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 10:45:11.504935 16052 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:11.497951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:34713 (local address 127.0.0.1:35632)
0927 10:45:11.498253 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:11.498268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:11.498385 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:11.498696 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:11.498699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:11.498706 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:45:11.498824 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:11.498830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.499587 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.499590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:11.500577 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:11.500587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.501023 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.501027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:11.501116 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.501144 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:45:11.502883 (+ 1739us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 10:45:11.504761 (+ 1878us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.96.254:34713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1690,"thread_start_us":109,"threads_started":1}
I20260927 10:45:11.505484 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 15956
E20260927 10:45:11.515870 16053 webserver.cc:533] Webserver: sslize failed:
E20260927 10:45:11.515964 16053 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T10:45:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (306 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T10:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:11Z Disabled control of system clock
I20260927 10:45:11.553638 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43615
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43615
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/testchainca.cert with env {}
W20260927 10:45:11.661600 16057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:11.661870 16057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:11.661937 16057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:11.665889 16057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:11.665943 16057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:11.665959 16057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:11.665977 16057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:11.665992 16057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 10:45:11.666047 16057 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 10:45:11.666074 16057 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 10:45:11.671375 16057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:11.672391 16057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:11.673403 16057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:11.681646 16062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:11.681648 16065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:11.681726 16063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:11.681933 16057 server_base.cc:1034] running on GCE node
I20260927 10:45:11.682619 16057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:11.683749 16057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:11.684922 16057 hybrid_clock.cc:648] HybridClock initialized: now 1790505911684910 us; error 27 us; skew 500 ppm
I20260927 10:45:11.686883 16057 webserver.cc:460] Webserver started at http://127.10.96.254:40567/ using document root <none> and password file <none>
I20260927 10:45:11.687460 16057 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:11.687520 16057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:11.687765 16057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:11.689865 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "c87de57a214a41fb8014d64b7fd27714"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "ec99a1bccaf933fef5f3372579b565db"
server_key_iv: "16d13a07f78e7f069d6425daeb013bc2"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.690433 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "c87de57a214a41fb8014d64b7fd27714"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "ec99a1bccaf933fef5f3372579b565db"
server_key_iv: "16d13a07f78e7f069d6425daeb013bc2"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.693970 16057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:45:11.696321 16071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:11.697598 16057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:11.697728 16057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "c87de57a214a41fb8014d64b7fd27714"
format_stamp: "Formatted at 2026-09-27 10:45:11 on dist-test-slave-wmg0"
server_key: "ec99a1bccaf933fef5f3372579b565db"
server_key_iv: "16d13a07f78e7f069d6425daeb013bc2"
server_key_version: "encryptionkey@0"
I20260927 10:45:11.697837 16057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:11.714208 16057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:11.714864 16057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:11.715058 16057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:11.722084 16057 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43615
I20260927 10:45:11.722095 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43615 every 8 connection(s)
I20260927 10:45:11.723107 16057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:11.725914 16124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:11.728953 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16057
I20260927 10:45:11.729074 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:11.729373 10627 external_mini_cluster.cc:1428] Setting key c6b38b96e0d319d4dfd91d0f539f4ff1
I20260927 10:45:11.731669 16124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714: Bootstrap starting.
I20260927 10:45:11.733815 16124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:11.734506 16124 log.cc:824] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:11.736290 16124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714: No bootstrap required, opened a new log
I20260927 10:45:11.739745 16124 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87de57a214a41fb8014d64b7fd27714" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43615 } }
I20260927 10:45:11.739977 16124 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:11.740067 16124 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87de57a214a41fb8014d64b7fd27714, State: Initialized, Role: FOLLOWER
I20260927 10:45:11.740520 16124 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87de57a214a41fb8014d64b7fd27714" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43615 } }
I20260927 10:45:11.740665 16124 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:11.740734 16124 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:11.740859 16124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:11.741665 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:11.730716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.10.96.254:43615)
0927 10:45:11.731354 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:11.731370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:11.731403 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:45:11.732035 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:11.732053 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:11.732449 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:11.732774 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:11.733211 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.734295 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.735202 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.735718 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.735900 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.739717 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:11.739731 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:11.739743 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:11.740226 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:11.740287 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:11.740308 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:11.740416 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:11.740564 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 10:45:11.740857 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":133,"threads_started":1}
I20260927 10:45:11.741812 16124 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87de57a214a41fb8014d64b7fd27714" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43615 } }
I20260927 10:45:11.742137 16124 leader_election.cc:302] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87de57a214a41fb8014d64b7fd27714; no voters:
I20260927 10:45:11.742414 16124 leader_election.cc:288] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:11.742514 16129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:11.742709 16129 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [term 1 LEADER]: Becoming Leader. State: Replica: c87de57a214a41fb8014d64b7fd27714, State: Running, Role: LEADER
I20260927 10:45:11.743018 16129 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87de57a214a41fb8014d64b7fd27714" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43615 } }
I20260927 10:45:11.743643 16124 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:11.744622 16130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87de57a214a41fb8014d64b7fd27714. Latest consensus state: current_term: 1 leader_uuid: "c87de57a214a41fb8014d64b7fd27714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87de57a214a41fb8014d64b7fd27714" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43615 } } }
I20260927 10:45:11.744817 16130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:11.744825 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:45:11.744686 16131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87de57a214a41fb8014d64b7fd27714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87de57a214a41fb8014d64b7fd27714" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43615 } } }
I20260927 10:45:11.745666 16131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:11.746546 16138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:11.749161 16138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:11.754309 16138 catalog_manager.cc:1347] Generated new cluster ID: f54546fe790745138df69fd6e595f906
I20260927 10:45:11.754370 16138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:11.785168 16138 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:11.786191 16138 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:11.793906 16138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87de57a214a41fb8014d64b7fd27714: Generated new TSK 0
I20260927 10:45:11.794682 16138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:11.854168 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:11.803029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.10.96.254:43615)
0927 10:45:11.803189 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:11.803194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:11.803395 (+ 201us) server_negotiation.cc:407] Connection header received
0927 10:45:11.803523 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:11.803525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:11.803572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:11.803662 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0927 10:45:11.803913 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.804441 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.805682 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.805910 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.805996 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.806323 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 10:45:11.853798 (+ 47475us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:45:11.854050 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37832: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:45:11.854271 16127 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37832: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 10:45:11.854758 16153 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:11.802984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43615 (local address 127.0.0.1:37832)
0927 10:45:11.803283 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:11.803298 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:11.803415 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:11.803681 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:11.803684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:11.803692 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 10:45:11.803812 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:11.803818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.804546 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.804550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:11.805553 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:11.805564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:11.806035 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:11.806039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:11.806133 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:11.806164 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 10:45:11.854043 (+ 47879us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 10:45:11.854267 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.96.254:43615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":120,"threads_started":1}
I20260927 10:45:11.855423 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16057
2026-09-27T10:45:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (329 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:12 dist-test-slave-wmg0 krb5kdc[16165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:12 dist-test-slave-wmg0 krb5kdc[16165](info): set up 2 sockets
Sep 27 10:45:12 dist-test-slave-wmg0 krb5kdc[16165](info): commencing operation
krb5kdc: starting...
W20260927 10:45:13.121965 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:13.240444 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:34619
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:34619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:13.359921 16178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:13.360183 16178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:13.360221 16178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:13.364347 16178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:13.364408 16178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:13.364426 16178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:13.364444 16178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:13.364499 16178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:13.369828 16178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:34619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:13.370859 16178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:13.371878 16178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:13.380206 16186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:13.380276 16178 server_base.cc:1034] running on GCE node
W20260927 10:45:13.380213 16183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:13.380213 16184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:13.381090 16178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:13.382205 16178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:13.383391 16178 hybrid_clock.cc:648] HybridClock initialized: now 1790505913383373 us; error 30 us; skew 500 ppm
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:13.386111 16178 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:13.386958 16178 webserver.cc:460] Webserver started at http://127.10.96.254:40095/ using document root <none> and password file <none>
I20260927 10:45:13.387509 16178 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:13.387599 16178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:13.387840 16178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:13.389875 16178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "90e69a7d49d543cf8cb20c1b213cda2e"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "07c3c0218d6bce95b2102dc9a21be87e"
server_key_iv: "a4b997d88ed3eb23416cd2c53c949047"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.390417 16178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "90e69a7d49d543cf8cb20c1b213cda2e"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "07c3c0218d6bce95b2102dc9a21be87e"
server_key_iv: "a4b997d88ed3eb23416cd2c53c949047"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.393937 16178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:13.396302 16193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.397627 16178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:13.397778 16178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "90e69a7d49d543cf8cb20c1b213cda2e"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "07c3c0218d6bce95b2102dc9a21be87e"
server_key_iv: "a4b997d88ed3eb23416cd2c53c949047"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.397886 16178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:13.407884 16178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:13.410741 16178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:13.410936 16178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:13.418092 16178 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:34619
I20260927 10:45:13.418114 16245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:34619 every 8 connection(s)
I20260927 10:45:13.419001 16178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:13.421624 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:13.426311 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16178
I20260927 10:45:13.426411 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:13.426678 10627 external_mini_cluster.cc:1428] Setting key 2de9ea0ba741e4bf983a07e38831c254
I20260927 10:45:13.427100 16246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e: Bootstrap starting.
I20260927 10:45:13.429414 16246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:13.430430 16246 log.cc:824] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:13.432613 16246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e: No bootstrap required, opened a new log
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:13.436887 16246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34619 } }
I20260927 10:45:13.437141 16246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:13.437263 16246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e69a7d49d543cf8cb20c1b213cda2e, State: Initialized, Role: FOLLOWER
I20260927 10:45:13.437757 16246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34619 } }
I20260927 10:45:13.437909 16246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:13.437995 16246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:13.438090 16246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:13.438944 16246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34619 } }
I20260927 10:45:13.439291 16246 leader_election.cc:302] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e69a7d49d543cf8cb20c1b213cda2e; no voters:
I20260927 10:45:13.439611 16246 leader_election.cc:288] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:13.439677 16251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:13.440073 16251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [term 1 LEADER]: Becoming Leader. State: Replica: 90e69a7d49d543cf8cb20c1b213cda2e, State: Running, Role: LEADER
I20260927 10:45:13.440371 16251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34619 } }
I20260927 10:45:13.440938 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:13.428055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.10.96.254:34619)
0927 10:45:13.428481 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:13.428505 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:13.428539 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:45:13.429135 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:13.429151 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:13.429521 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:13.429845 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:13.430753 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.431911 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.432795 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.433240 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.433417 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:13.436002 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:13.436015 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:13.436028 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:13.436053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:13.438045 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:13.438475 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:13.438481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:13.438486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:13.438604 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:13.438882 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:13.438887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:13.438888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:13.439329 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:13.439515 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:13.439810 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 10:45:13.440098 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260927 10:45:13.440972 16246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:13.441562 16251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e69a7d49d543cf8cb20c1b213cda2e. Latest consensus state: current_term: 1 leader_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34619 } } }
I20260927 10:45:13.441754 16251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:13.442024 16251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e69a7d49d543cf8cb20c1b213cda2e" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34619 } } }
I20260927 10:45:13.442165 16251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e [sys.catalog]: This master's current role is: LEADER
W20260927 10:45:13.445674 16267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:45:13.445739 16267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:45:13.445844 16258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:13.446522 16258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:13.451578 16258 catalog_manager.cc:1347] Generated new cluster ID: e48735d25a9f4ca9bfc2e6982a059f18
I20260927 10:45:13.451642 16258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:13.469437 16258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:13.470749 16258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:13.477031 16258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e69a7d49d543cf8cb20c1b213cda2e: Generated new TSK 0
I20260927 10:45:13.477895 16258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:45:13.512681 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:34619
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:13.623059 16272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:13.623334 16272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:13.623378 16272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:13.627333 16272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:13.627391 16272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:13.627449 16272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:13.632655 16272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:13.633670 16272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:13.634600 16272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:13.644254 16277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:13.644250 16278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:13.644250 16280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:13.644362 16272 server_base.cc:1034] running on GCE node
I20260927 10:45:13.644979 16272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:13.645615 16272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:13.646777 16272 hybrid_clock.cc:648] HybridClock initialized: now 1790505913646757 us; error 31 us; skew 500 ppm
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:13.649436 16272 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:45:13.650276 16272 webserver.cc:460] Webserver started at http://127.10.96.193:43329/ using document root <none> and password file <none>
I20260927 10:45:13.650775 16272 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:13.650831 16272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:13.650987 16272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:13.652865 16272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "7af8859e41f04dab875100c367d53036"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "0c668afbdd30a84572b2f38dc6e4a8c9"
server_key_iv: "2388435b6275a5dc8c34b3f9e8bb6018"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.653339 16272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "7af8859e41f04dab875100c367d53036"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "0c668afbdd30a84572b2f38dc6e4a8c9"
server_key_iv: "2388435b6275a5dc8c34b3f9e8bb6018"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.656843 16272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:45:13.659336 16287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.660706 16272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:45:13.660805 16272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "7af8859e41f04dab875100c367d53036"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "0c668afbdd30a84572b2f38dc6e4a8c9"
server_key_iv: "2388435b6275a5dc8c34b3f9e8bb6018"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.660892 16272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:13.684999 16272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:13.688436 16272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:13.688638 16272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:13.689234 16272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:13.690094 16272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:13.690142 16272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.690181 16272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:13.690207 16272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.698642 16272 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:36019
I20260927 10:45:13.698663 16400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:36019 every 8 connection(s)
I20260927 10:45:13.699563 16272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:13.709146 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16272
I20260927 10:45:13.709280 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:13.709556 10627 external_mini_cluster.cc:1428] Setting key 264ca0d1f71a826f5898d9a7ecce82e3
I20260927 10:45:13.713738 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:13.701636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:34491 (local address 127.10.96.254:34619)
0927 10:45:13.701777 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:13.701781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:13.702644 (+ 863us) server_negotiation.cc:407] Connection header received
0927 10:45:13.703979 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:13.703983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:13.704038 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:13.704118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:13.705269 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.705864 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.706977 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.707211 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.707322 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:13.709773 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:13.709778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:13.709781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:13.709807 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:13.711466 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:13.712046 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:13.712052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:13.712056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:13.712147 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:13.712465 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:13.712471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:13.712475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:13.712714 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:13.712823 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:13.713515 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 10:45:13.713623 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:45:13.714912 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:13.702057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:34619 (local address 127.10.96.193:34491)
0927 10:45:13.702498 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:13.702539 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:13.703757 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:13.704283 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:13.704290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:13.704642 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:13.705116 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:13.705129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.705983 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.705987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:13.706845 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:13.706853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.707357 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.707361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:13.707527 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:13.708067 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:13.708082 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:13.709597 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:13.711631 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:13.711636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:13.711647 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:13.711920 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:13.712243 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:13.712245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:13.712246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:13.712376 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:13.712853 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:13.712861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:13.713197 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 10:45:13.713423 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 10:45:13.713829 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260927 10:45:13.716156 16401 heartbeater.cc:347] Connected to a master server at 127.10.96.254:34619
I20260927 10:45:13.716410 16401 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:13.716897 16401 heartbeater.cc:511] Master 127.10.96.254:34619 requested a full tablet report, sending...
I20260927 10:45:13.718559 16210 ts_manager.cc:194] Registered new tserver with Master: 7af8859e41f04dab875100c367d53036 (127.10.96.193:36019)
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:45:13.720394 16210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:34491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:45:13.738559 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:34619
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:13.848734 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:13.849036 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:13.849110 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:13.853209 16406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:13.853299 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:13.853390 16406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:13.858683 16406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:13.859760 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:13.860780 16406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:13.870669 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:13.870667 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:13.870674 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:13.870724 16406 server_base.cc:1034] running on GCE node
I20260927 10:45:13.871284 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:13.871932 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:13.873082 16406 hybrid_clock.cc:648] HybridClock initialized: now 1790505913873066 us; error 28 us; skew 500 ppm
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:13.875847 16406 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:45:13.876716 16406 webserver.cc:460] Webserver started at http://127.10.96.194:41693/ using document root <none> and password file <none>
I20260927 10:45:13.877280 16406 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:13.877365 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:13.877635 16406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:13.879624 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "a689a2e8ec2c4f54a260cf10da85c346"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "5cc7df7ef2708b1ea3e6521905618bc8"
server_key_iv: "1cd98a52406a3c630e10836b7bee1573"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.880182 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "a689a2e8ec2c4f54a260cf10da85c346"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "5cc7df7ef2708b1ea3e6521905618bc8"
server_key_iv: "1cd98a52406a3c630e10836b7bee1573"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.883883 16406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:45:13.886392 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.887687 16406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:13.887845 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a689a2e8ec2c4f54a260cf10da85c346"
format_stamp: "Formatted at 2026-09-27 10:45:13 on dist-test-slave-wmg0"
server_key: "5cc7df7ef2708b1ea3e6521905618bc8"
server_key_iv: "1cd98a52406a3c630e10836b7bee1573"
server_key_version: "encryptionkey@0"
I20260927 10:45:13.887959 16406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:13.905159 16406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:13.908202 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:13.908409 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:13.908981 16406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:13.909924 16406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:13.909989 16406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.910053 16406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:13.910097 16406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:13.919091 16406 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:40715
I20260927 10:45:13.919111 16534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:40715 every 8 connection(s)
I20260927 10:45:13.919970 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:13.924894 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16406
I20260927 10:45:13.925014 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:13.925304 10627 external_mini_cluster.cc:1428] Setting key 76edf554d85aa13489cc78332f4ba1e2
Sep 27 10:45:13 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:45:13.935189 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:13.921672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:60819 (local address 127.10.96.254:34619)
0927 10:45:13.921828 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:13.921832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:13.922609 (+ 777us) server_negotiation.cc:407] Connection header received
0927 10:45:13.923905 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:13.923908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:13.923957 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:13.924040 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:13.925637 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.926204 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.927891 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.928102 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.928185 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:13.930923 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:13.930926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:13.930928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:13.930951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:13.932841 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:13.933399 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:13.933404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:13.933406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:13.933540 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:13.933936 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:13.933942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:13.933946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:13.934209 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:13.934325 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:13.934880 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 10:45:13.935018 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 10:45:13.936115 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:13.922037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:34619 (local address 127.10.96.194:60819)
0927 10:45:13.922459 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:13.922504 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:13.923692 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:13.924280 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:13.924287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:13.924754 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:13.925480 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:13.925501 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.926349 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.926352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:13.927768 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:13.927782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:13.928281 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:13.928288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:13.928521 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:13.929182 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:13.929198 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:13.930739 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:13.932985 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:13.932991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:13.933003 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:13.933274 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:13.933650 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:13.933653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:13.933654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:13.933815 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:13.934337 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:13.934342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:13.934593 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 10:45:13.934763 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 10:45:13.935114 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:45:13.937668 16535 heartbeater.cc:347] Connected to a master server at 127.10.96.254:34619
I20260927 10:45:13.937937 16535 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:13.938539 16535 heartbeater.cc:511] Master 127.10.96.254:34619 requested a full tablet report, sending...
I20260927 10:45:13.939781 16210 ts_manager.cc:194] Registered new tserver with Master: a689a2e8ec2c4f54a260cf10da85c346 (127.10.96.194:40715)
I20260927 10:45:13.940641 16210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:60819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:45:13.952486 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:34619
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:14.062273 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:14.062529 16540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:14.062611 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:14.066769 16540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:14.066828 16540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:14.066917 16540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:14.073171 16540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:34619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:14.074265 16540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:14.075284 16540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:14.084692 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:14.084683 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:14.084681 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:14.084911 16540 server_base.cc:1034] running on GCE node
I20260927 10:45:14.085450 16540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:14.086139 16540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:14.087302 16540 hybrid_clock.cc:648] HybridClock initialized: now 1790505914087273 us; error 42 us; skew 500 ppm
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.090178 16540 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:14.090983 16540 webserver.cc:460] Webserver started at http://127.10.96.195:39205/ using document root <none> and password file <none>
I20260927 10:45:14.091483 16540 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:14.091538 16540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:14.091765 16540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:14.093762 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "ff42554c46044643a33c93ac2934c4c2"
format_stamp: "Formatted at 2026-09-27 10:45:14 on dist-test-slave-wmg0"
server_key: "f21ca28886c06ae864cb55c048773001"
server_key_iv: "8a5f999ea45d8976802eaf9a4c569819"
server_key_version: "encryptionkey@0"
I20260927 10:45:14.094254 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "ff42554c46044643a33c93ac2934c4c2"
format_stamp: "Formatted at 2026-09-27 10:45:14 on dist-test-slave-wmg0"
server_key: "f21ca28886c06ae864cb55c048773001"
server_key_iv: "8a5f999ea45d8976802eaf9a4c569819"
server_key_version: "encryptionkey@0"
I20260927 10:45:14.097827 16540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:45:14.100231 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:14.101715 16540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:14.101982 16540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "ff42554c46044643a33c93ac2934c4c2"
format_stamp: "Formatted at 2026-09-27 10:45:14 on dist-test-slave-wmg0"
server_key: "f21ca28886c06ae864cb55c048773001"
server_key_iv: "8a5f999ea45d8976802eaf9a4c569819"
server_key_version: "encryptionkey@0"
I20260927 10:45:14.102102 16540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:14.123546 16540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:14.127244 16540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:14.127452 16540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:14.128063 16540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:14.128976 16540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:14.129047 16540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:14.129108 16540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:14.129155 16540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:14.137804 16540 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:34641
I20260927 10:45:14.137837 16668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:34641 every 8 connection(s)
I20260927 10:45:14.138774 16540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:14.148926 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16540
I20260927 10:45:14.149055 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:14.149435 10627 external_mini_cluster.cc:1428] Setting key d83688a2acea40c24ee17fea625d1a2b
I20260927 10:45:14.153112 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:14.140573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:60903 (local address 127.10.96.254:34619)
0927 10:45:14.140783 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.140788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.141725 (+ 937us) server_negotiation.cc:407] Connection header received
0927 10:45:14.143257 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.143261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.143320 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.143428 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.144581 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.145114 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.146306 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.146523 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.146633 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:14.149266 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:14.149269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:14.149271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:14.149294 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:14.151038 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.151657 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.151660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.151662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.151740 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.152076 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.152081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.152082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.152261 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:14.152363 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:14.152838 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 10:45:14.152951 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:45:14.153965 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:14.140997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:34619 (local address 127.10.96.195:60903)
0927 10:45:14.141547 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:14.141604 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:14.143040 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:14.143597 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:14.143604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:14.143957 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:14.144431 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:14.144444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.145238 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.145241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:14.146179 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:14.146188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.146672 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.146675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:14.146838 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:14.147451 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:14.147466 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:14.149081 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:14.151143 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:14.151149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:14.151163 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:14.151519 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:14.151834 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:14.151840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:14.151843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:14.151980 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:14.152371 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:14.152377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:14.152614 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 10:45:14.152801 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:45:14.153114 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":100,"threads_started":1}
I20260927 10:45:14.155055 16669 heartbeater.cc:347] Connected to a master server at 127.10.96.254:34619
I20260927 10:45:14.155308 16669 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:14.155776 16669 heartbeater.cc:511] Master 127.10.96.254:34619 requested a full tablet report, sending...
I20260927 10:45:14.156785 16210 ts_manager.cc:194] Registered new tserver with Master: ff42554c46044643a33c93ac2934c4c2 (127.10.96.195:34641)
I20260927 10:45:14.157310 16210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:60903
I20260927 10:45:14.163517 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/test-admin.keytab.
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.179523 10627 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 10:45:14.182344 16676 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:14.180841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.10.96.193:36019)
0927 10:45:14.181157 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.181172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.181195 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:45:14.181269 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.181274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.181520 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.181659 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.182145 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37136: BlockingRecv error: recv got EOF from 127.0.0.1:37136 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
W20260927 10:45:14.182355 10627 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.183490 10627 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.184366 10627 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:45:14.193416 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:14.185197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.10.96.193:36019)
0927 10:45:14.185330 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.185333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.185347 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:14.185534 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.185537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.185573 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.185639 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.186122 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.186898 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.187685 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.188064 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.188141 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:14.190300 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:14.190308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:14.190320 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:14.190340 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:14.191861 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.192303 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.192308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.192309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.192397 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.192645 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.192648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.192649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.192867 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:14.193027 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:14.193163 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 10:45:14.193275 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 10:45:14.196938 16676 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:14.196031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.10.96.193:36019)
0927 10:45:14.196146 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.196150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.196253 (+ 103us) server_negotiation.cc:407] Connection header received
0927 10:45:14.196354 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.196357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.196414 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.196494 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.196856 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37152: BlockingRecv error: recv got EOF from 127.0.0.1:37152 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 10:45:14.197239 10627 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc' of type FILE
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.198275 10627 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.199133 10627 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:45:14.207572 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:14.199879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.10.96.193:36019)
0927 10:45:14.199996 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.200001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.200114 (+ 113us) server_negotiation.cc:407] Connection header received
0927 10:45:14.200200 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.200202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.200233 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.200294 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.200884 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.201452 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.202253 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.202476 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.202549 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:14.204595 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:14.204597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:14.204599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:14.204616 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:14.206025 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.206380 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.206383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.206384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.206453 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.206753 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.206756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.206757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.206909 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:14.207031 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:14.207279 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 10:45:14.207432 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:45:14.210139 16676 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:14.209163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.10.96.193:36019)
0927 10:45:14.209281 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.209284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.209499 (+ 215us) server_negotiation.cc:407] Connection header received
0927 10:45:14.209605 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.209607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.209642 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.209710 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.210053 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37172: BlockingRecv error: recv got EOF from 127.0.0.1:37172 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 10:45:14.210469 10627 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790505884435945-10627-0/krb5kdc/krb5cc)
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.211452 10627 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:14.212369 10627 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:45:14.220697 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:14.213189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.10.96.193:36019)
0927 10:45:14.213295 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:14.213298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:14.213331 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:45:14.213432 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:14.213434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:14.213550 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:14.213614 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:14.214198 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.214665 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.215451 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:14.215737 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:14.215819 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:14.217906 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:14.217909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:14.217910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:14.217929 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:14.219290 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.219652 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.219654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.219655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.219717 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:14.219936 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:14.219938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:14.219939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:14.220077 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:14.220166 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:14.220426 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 10:45:14.220502 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 10:45:14.221707 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16272
I20260927 10:45:14.227567 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16406
I20260927 10:45:14.233328 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16540
I20260927 10:45:14.238255 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16178
2026-09-27T10:45:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2144 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16689](info): set up 2 sockets
Sep 27 10:45:14 dist-test-slave-wmg0 krb5kdc[16689](info): commencing operation
krb5kdc: starting...
W20260927 10:45:15.270921 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:15 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab.
Entry for principal oryx/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab.
I20260927 10:45:15.386411 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:32875
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab
--principal=oryx/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:32875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:15.492002 16702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:15.492261 16702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:15.492300 16702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:15.496342 16702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:15.496400 16702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:15.496420 16702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:15.496436 16702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:15.496475 16702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:15.501588 16702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:32875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab
--principal=oryx/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:15.502605 16702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:15.503600 16702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:15.511786 16708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:15.511811 16707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:15.511967 16702 server_base.cc:1034] running on GCE node
W20260927 10:45:15.511821 16710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:15.512732 16702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:15.513877 16702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:15.515064 16702 hybrid_clock.cc:648] HybridClock initialized: now 1790505915515034 us; error 43 us; skew 500 ppm
Sep 27 10:45:15 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:15.518035 16702 init.cc:377] Logged in from keytab as oryx/127.10.96.254@KRBTEST.COM (short username oryx)
I20260927 10:45:15.518889 16702 webserver.cc:460] Webserver started at http://127.10.96.254:35603/ using document root <none> and password file <none>
I20260927 10:45:15.519413 16702 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:15.519471 16702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:15.519630 16702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:15.521697 16702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "2e9a098562700392181d4d87811551d7"
server_key_iv: "c68f017c8f58343c577e7877b2f16ac4"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.522204 16702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "2e9a098562700392181d4d87811551d7"
server_key_iv: "c68f017c8f58343c577e7877b2f16ac4"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.525748 16702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:15.528059 16717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:15.529714 16702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:15.529834 16702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "2e9a098562700392181d4d87811551d7"
server_key_iv: "c68f017c8f58343c577e7877b2f16ac4"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.529939 16702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:15.539932 16702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:15.543268 16702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:15.543479 16702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:15.549983 16702 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:32875
I20260927 10:45:15.550235 16769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:32875 every 8 connection(s)
I20260927 10:45:15.550969 16702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:15.551916 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16702
I20260927 10:45:15.552039 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:15.552362 10627 external_mini_cluster.cc:1428] Setting key 04b023af485a29b8323767adab3f7bfd
I20260927 10:45:15.554255 16770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:15.560883 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Bootstrap starting.
Sep 27 10:45:15 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:15.563642 16770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:15.564508 16770 log.cc:824] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:15.566380 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: No bootstrap required, opened a new log
I20260927 10:45:15.567263 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:15.553753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.10.96.254:32875)
0927 10:45:15.554157 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:15.554174 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:15.554215 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:45:15.555031 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:15.555052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:15.555454 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:15.555809 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:15.556466 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:15.557995 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:15.558828 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:15.559503 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:15.559667 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:15.562387 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:15.562392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:15.562407 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:15.562437 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:15.564484 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:15.564959 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:15.564969 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:15.564973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:15.565078 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:15.565379 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:15.565384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:15.565386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:15.565821 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:15.566019 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:15.566290 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:45:15.566594 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260927 10:45:15.569564 16770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } }
I20260927 10:45:15.569851 16770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:15.569954 16770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2231c3f1a56a41c0965b1e7bf3b3ad02, State: Initialized, Role: FOLLOWER
I20260927 10:45:15.570398 16770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } }
I20260927 10:45:15.570518 16770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:15.570581 16770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:15.570674 16770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:15.571542 16770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } }
I20260927 10:45:15.571870 16770 leader_election.cc:302] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2231c3f1a56a41c0965b1e7bf3b3ad02; no voters:
I20260927 10:45:15.572170 16770 leader_election.cc:288] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:15.572412 16775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:15.572615 16775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 LEADER]: Becoming Leader. State: Replica: 2231c3f1a56a41c0965b1e7bf3b3ad02, State: Running, Role: LEADER
I20260927 10:45:15.572886 16775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } }
I20260927 10:45:15.573290 16770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:15.574534 16776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } }
I20260927 10:45:15.574652 16776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:15.575012 16784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:15.574975 16777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2231c3f1a56a41c0965b1e7bf3b3ad02. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } }
I20260927 10:45:15.575412 16777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:15.578888 16784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:15.584260 16784 catalog_manager.cc:1347] Generated new cluster ID: 77ca7f35a69446c39f57a0db1f742941
I20260927 10:45:15.584359 16784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:15.591019 16784 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:15.591928 16784 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:15.606331 16784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Generated new TSK 0
I20260927 10:45:15.607101 16784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab.
Entry for principal oryx/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab.
I20260927 10:45:15.638701 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32875
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:15.749518 16796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:15.749770 16796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:15.749845 16796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:15.753854 16796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:15.753909 16796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:15.753993 16796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:15.759143 16796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:15.760303 16796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:15.761390 16796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:45:15.771183 16796 server_base.cc:1034] running on GCE node
W20260927 10:45:15.771127 16804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:15.771127 16801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:15.771127 16802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:15.771797 16796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:15.772382 16796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:15.773533 16796 hybrid_clock.cc:648] HybridClock initialized: now 1790505915773515 us; error 28 us; skew 500 ppm
Sep 27 10:45:15 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:15.776350 16796 init.cc:377] Logged in from keytab as oryx/127.10.96.193@KRBTEST.COM (short username oryx)
I20260927 10:45:15.777226 16796 webserver.cc:460] Webserver started at http://127.10.96.193:43001/ using document root <none> and password file <none>
I20260927 10:45:15.777845 16796 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:15.777933 16796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:15.778162 16796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:15.780172 16796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "d618480d3ce74fd1b605fa2c691ff70c"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "4a95ac106efcd9fad085ee248c0e7cc1"
server_key_iv: "3285354c03e6a2e8b2f41c3fd2d3d517"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.780715 16796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "d618480d3ce74fd1b605fa2c691ff70c"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "4a95ac106efcd9fad085ee248c0e7cc1"
server_key_iv: "3285354c03e6a2e8b2f41c3fd2d3d517"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.784327 16796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:45:15.786777 16811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:15.787945 16796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:15.788084 16796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "d618480d3ce74fd1b605fa2c691ff70c"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "4a95ac106efcd9fad085ee248c0e7cc1"
server_key_iv: "3285354c03e6a2e8b2f41c3fd2d3d517"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.788193 16796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:15.803797 16796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:15.807018 16796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:15.807341 16796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:15.807941 16796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:15.808759 16796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:15.808833 16796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:15.808897 16796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:15.808946 16796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:15.818411 16796 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:35571
I20260927 10:45:15.818429 16924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:35571 every 8 connection(s)
I20260927 10:45:15.819281 16796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:15.824764 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16796
I20260927 10:45:15.824913 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:15.825186 10627 external_mini_cluster.cc:1428] Setting key 60bf863a44d6f3d0faafc40ea62456eb
Sep 27 10:45:15 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:15.833853 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:15.821052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:36541 (local address 127.10.96.254:32875)
0927 10:45:15.821185 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:15.821195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:15.822159 (+ 964us) server_negotiation.cc:407] Connection header received
0927 10:45:15.823518 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:15.823522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:15.823581 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:15.823658 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:15.824836 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:15.825592 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:15.826759 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:15.826998 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:15.827106 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:15.829930 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:15.829936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:15.829943 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:15.829975 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:15.831760 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:15.832244 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:15.832250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:15.832255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:15.832347 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:15.832659 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:15.832662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:15.832664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:15.832889 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:15.833001 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:15.833533 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 10:45:15.833669 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for oryx/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:45:15.834692 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:15.821514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:36541)
0927 10:45:15.822004 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:15.822051 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:15.823306 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:15.823836 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:15.823842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:15.824193 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:15.824676 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:15.824690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:15.825723 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:15.825726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:15.826600 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:15.826609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:15.827141 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:15.827145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:15.827323 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:15.827974 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:15.828000 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:15.829762 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:15.831877 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:15.831881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:15.831893 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:15.832137 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:15.832439 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:15.832441 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:15.832442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:15.832564 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:15.832996 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:15.833001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:15.833271 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 10:45:15.833434 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 10:45:15.833757 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260927 10:45:15.835961 16925 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32875
I20260927 10:45:15.836192 16925 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:15.836692 16925 heartbeater.cc:511] Master 127.10.96.254:32875 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.194@KRBTEST.COM" created.
I20260927 10:45:15.838240 16734 ts_manager.cc:194] Registered new tserver with Master: d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571)
I20260927 10:45:15.839356 16734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:36541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab.
Entry for principal oryx/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab.
I20260927 10:45:15.851683 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32875
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:15.959758 16930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:15.960083 16930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:15.960162 16930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:15.964184 16930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:15.964284 16930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:15.964378 16930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:15.969579 16930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:15.970613 16930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:15.971657 16930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:15.981190 16935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:15.981171 16938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:15.981220 16930 server_base.cc:1034] running on GCE node
W20260927 10:45:15.981175 16936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:15.981802 16930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:15.982389 16930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:15.983557 16930 hybrid_clock.cc:648] HybridClock initialized: now 1790505915983541 us; error 28 us; skew 500 ppm
Sep 27 10:45:15 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:15.986159 16930 init.cc:377] Logged in from keytab as oryx/127.10.96.194@KRBTEST.COM (short username oryx)
I20260927 10:45:15.987032 16930 webserver.cc:460] Webserver started at http://127.10.96.194:42995/ using document root <none> and password file <none>
I20260927 10:45:15.987581 16930 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:15.987668 16930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:15.987891 16930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:15.989903 16930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "1dbbf06c99e94029b60d7f9a26fda666"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "1394aecd7d2edb303ea437145e21b18b"
server_key_iv: "d766537064a0fd2765329517110c24af"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.990460 16930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "1dbbf06c99e94029b60d7f9a26fda666"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "1394aecd7d2edb303ea437145e21b18b"
server_key_iv: "d766537064a0fd2765329517110c24af"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.993986 16930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:15.996289 16945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:15.997574 16930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:15.997697 16930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "1dbbf06c99e94029b60d7f9a26fda666"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "1394aecd7d2edb303ea437145e21b18b"
server_key_iv: "d766537064a0fd2765329517110c24af"
server_key_version: "encryptionkey@0"
I20260927 10:45:15.997803 16930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.013196 16930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:16.016028 16930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:16.016242 16930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:16.016819 16930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:16.017711 16930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:16.017782 16930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.017851 16930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:16.017892 16930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.026777 16930 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:44993
I20260927 10:45:16.026825 17058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:44993 every 8 connection(s)
I20260927 10:45:16.027693 16930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:16.037740 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 16930
I20260927 10:45:16.037859 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:16.038393 10627 external_mini_cluster.cc:1428] Setting key 39be84e75704f11a148e1d3e740b9ba1
I20260927 10:45:16.041286 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.029483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:44939 (local address 127.10.96.254:32875)
0927 10:45:16.029607 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.029610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.030466 (+ 856us) server_negotiation.cc:407] Connection header received
0927 10:45:16.031790 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.031793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.031854 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.031934 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.033058 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.033693 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.034840 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.035064 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.035177 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.037609 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.037612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.037614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.037638 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.039223 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.039802 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.039808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.039814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.039886 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.040236 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.040240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.040242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.040419 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.040527 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.040995 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.041115 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:45:16.042268 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.029866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:44939)
0927 10:45:16.030310 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.030352 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.031580 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.032098 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.032105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.032456 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.032910 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.032923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.033822 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.033826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.034713 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.034722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.035212 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.035224 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.035384 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.035940 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.035956 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.037403 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.039328 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.039333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.039346 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.039670 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.039978 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.039983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.039985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.040122 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.040524 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.040529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.040769 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.040952 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.041255 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260927 10:45:16.043455 17059 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32875
I20260927 10:45:16.043700 17059 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:16.044157 17059 heartbeater.cc:511] Master 127.10.96.254:32875 requested a full tablet report, sending...
I20260927 10:45:16.045161 16734 ts_manager.cc:194] Registered new tserver with Master: 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993)
I20260927 10:45:16.045795 16734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:44939
WARNING: no policy specified for oryx/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab.
Entry for principal oryx/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab.
I20260927 10:45:16.064594 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32875
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:16.175526 17064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:16.175772 17064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:16.175846 17064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:16.180131 17064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:16.180183 17064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:16.180271 17064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:16.185441 17064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:16.186465 17064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:16.187466 17064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:16.196938 17070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.196942 17069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.196942 17072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:16.197157 17064 server_base.cc:1034] running on GCE node
I20260927 10:45:16.197722 17064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:16.198328 17064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:16.199476 17064 hybrid_clock.cc:648] HybridClock initialized: now 1790505916199445 us; error 43 us; skew 500 ppm
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:16.202208 17064 init.cc:377] Logged in from keytab as oryx/127.10.96.195@KRBTEST.COM (short username oryx)
I20260927 10:45:16.203082 17064 webserver.cc:460] Webserver started at http://127.10.96.195:38243/ using document root <none> and password file <none>
I20260927 10:45:16.203630 17064 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.203718 17064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.203945 17064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:16.205960 17064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "dfba3370ab7e43be9d07571f4d24e5cd"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "35320606e377eb6f277f7d43faae8313"
server_key_iv: "49b5896f1a319a4a0f150e34913b70aa"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.206511 17064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "dfba3370ab7e43be9d07571f4d24e5cd"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "35320606e377eb6f277f7d43faae8313"
server_key_iv: "49b5896f1a319a4a0f150e34913b70aa"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.209982 17064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:45:16.212231 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.213407 17064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:16.213560 17064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "dfba3370ab7e43be9d07571f4d24e5cd"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "35320606e377eb6f277f7d43faae8313"
server_key_iv: "49b5896f1a319a4a0f150e34913b70aa"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.213666 17064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.229115 17064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:16.232685 17064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:16.232899 17064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:16.233637 17064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:16.234504 17064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:16.234571 17064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.234643 17064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:16.234686 17064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.243757 17064 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:38787
I20260927 10:45:16.243780 17192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:38787 every 8 connection(s)
I20260927 10:45:16.244634 17064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:16.250885 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 17064
I20260927 10:45:16.251030 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:16.251320 10627 external_mini_cluster.cc:1428] Setting key 1f182c2cc95dc1450d555769d084a939
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:16.258749 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.246519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56975 (local address 127.10.96.254:32875)
0927 10:45:16.246688 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.246692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.247559 (+ 867us) server_negotiation.cc:407] Connection header received
0927 10:45:16.248888 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.248891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.248937 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.249020 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.250351 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.250872 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.252591 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.252896 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.253006 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.255440 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.255443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.255445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.255469 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.256951 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.257424 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.257426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.257427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.257547 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.257842 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.257844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.257845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.258020 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.258095 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.258539 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.258632 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 10:45:16.259572 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.246922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:56975)
0927 10:45:16.247404 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.247448 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.248656 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.249213 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.249242 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.249706 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.250190 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.250204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.251023 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.251035 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.252453 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.252464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.253057 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.253060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.253203 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.253855 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.253872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.255297 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.257052 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.257057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.257068 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.257329 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.257624 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.257626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.257627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.257754 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.258107 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.258112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.258345 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.258496 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.258730 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":109,"threads_started":1}
I20260927 10:45:16.260689 17193 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32875
I20260927 10:45:16.260922 17193 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:16.261389 17193 heartbeater.cc:511] Master 127.10.96.254:32875 requested a full tablet report, sending...
I20260927 10:45:16.262391 16734 ts_manager.cc:194] Registered new tserver with Master: dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787)
I20260927 10:45:16.263031 16734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:56975
I20260927 10:45:16.265213 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.253.keytab.
Entry for principal oryx/127.10.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.253.keytab.
I20260927 10:45:16.292531 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:42561
--webserver_interface=127.10.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.253.keytab
--principal=oryx/127.10.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:32875,127.10.96.253:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:16.403973 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:16.404270 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:16.404371 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:16.408524 17198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:16.408598 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:16.408635 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:16.408694 17198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:16.408733 17198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:16.413982 17198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:32875,127.10.96.253:42561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.253.keytab
--principal=oryx/127.10.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
W20260927 10:45:16.414214 17198 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.96.254:32875,127.10.96.253:42561'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 10:45:16.415066 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:16.416044 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:16.424391 17204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.424396 17203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.424401 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:16.424537 17198 server_base.cc:1034] running on GCE node
I20260927 10:45:16.425133 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:16.426245 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:16.427443 17198 hybrid_clock.cc:648] HybridClock initialized: now 1790505916427427 us; error 30 us; skew 500 ppm
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:16.430169 17198 init.cc:377] Logged in from keytab as oryx/127.10.96.253@KRBTEST.COM (short username oryx)
I20260927 10:45:16.431010 17198 webserver.cc:460] Webserver started at http://127.10.96.253:43013/ using document root <none> and password file <none>
I20260927 10:45:16.431565 17198 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.431653 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.431877 17198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:16.433979 17198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/instance:
uuid: "53234ec893754ca48c2936400df84c38"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "b83432d86bbf5b4c5fef0e67590bd756"
server_key_iv: "ce1b0df66d031fc1f5f77c14e0bcf8b6"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.434521 17198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal/instance:
uuid: "53234ec893754ca48c2936400df84c38"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "b83432d86bbf5b4c5fef0e67590bd756"
server_key_iv: "ce1b0df66d031fc1f5f77c14e0bcf8b6"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.438041 17198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:16.440276 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.441571 17198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:16.441701 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
uuid: "53234ec893754ca48c2936400df84c38"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "b83432d86bbf5b4c5fef0e67590bd756"
server_key_iv: "ce1b0df66d031fc1f5f77c14e0bcf8b6"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.441804 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.456518 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:16.459813 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:16.460026 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:16.466768 17198 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.253:42561
I20260927 10:45:16.467034 17265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.253:42561 every 8 connection(s)
I20260927 10:45:16.467741 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
I20260927 10:45:16.468647 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 17198
I20260927 10:45:16.470778 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:16.475327 17266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.253@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:16.483389 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.470176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.10.96.253:42561)
0927 10:45:16.470573 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.470590 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.470632 (+ 42us) server_negotiation.cc:407] Connection header received
0927 10:45:16.471458 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.471481 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.471885 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.472252 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.472863 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.474396 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.475421 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.476090 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.476254 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.479054 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.479058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.479067 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.479091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.480833 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.481241 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.481256 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.481265 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.481370 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.481672 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.481675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.481677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.482023 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.482257 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.482473 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.482714 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260927 10:45:16.485998 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.477232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:46860)
0927 10:45:16.477633 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.477688 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.477802 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.478097 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.478101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.478406 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.478584 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.478590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.479448 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.479451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.480078 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.480085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.480527 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.480530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.480648 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.481134 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.481146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.482435 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.484376 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.484383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.484387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.484681 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.484939 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.484941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.484943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.485029 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.485443 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.485447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.485597 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.485737 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.485866 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":164,"threads_started":1}
I20260927 10:45:16.486012 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.477289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.10.96.254:32875)
0927 10:45:16.477445 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.477451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.477774 (+ 323us) server_negotiation.cc:407] Connection header received
0927 10:45:16.477900 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.477903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.477954 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.478050 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.478710 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.479320 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.480204 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.480420 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.480526 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.482568 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.482573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.482576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.482604 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.484268 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.484777 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.484779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.484780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.484852 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.485114 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.485116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.485117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.485325 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.485438 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.485818 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.485906 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:45:16.487112 17266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.253@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
I20260927 10:45:16.494814 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.487607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.10.96.253:42561)
0927 10:45:16.487695 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.487700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.487761 (+ 61us) server_negotiation.cc:407] Connection header received
0927 10:45:16.487872 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.487876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.487915 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.487993 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.488678 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.489166 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.489941 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.490161 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.490265 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.492055 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.492059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.492061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.492088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.493221 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.493665 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.493671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.493673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.493747 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.494013 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.494016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.494018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.494167 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.494269 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.494498 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.494640 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":39}
I20260927 10:45:16.494814 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.487542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60042)
0927 10:45:16.487683 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.487698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.487782 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.488002 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.488005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.488152 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.488563 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.488569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.489285 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.489288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.489838 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.489846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.490291 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.490294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.490380 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.490746 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.490758 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.491929 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.493324 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.493330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.493335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.493578 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.493836 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.493841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.493843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.493920 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.494269 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.494272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.494349 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.494457 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.494586 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 10:45:16.498114 17266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Bootstrap starting.
I20260927 10:45:16.500313 17266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:16.501016 17266 log.cc:824] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:16.503036 17266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: No bootstrap required, opened a new log
I20260927 10:45:16.505874 17266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } }
I20260927 10:45:16.506068 17266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:16.506102 17266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53234ec893754ca48c2936400df84c38, State: Initialized, Role: FOLLOWER
I20260927 10:45:16.506455 17266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } }
I20260927 10:45:16.506867 17272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } } }
I20260927 10:45:16.506983 17272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:16.507516 17266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:45:16.512048 17283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:45:16.512130 17283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:45:16.513725 17198 master_runner.cc:428] Detected that this master 53234ec893754ca48c2936400df84c38 is joining an existing cluster
I20260927 10:45:16.513795 17198 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.96.253:42561
I20260927 10:45:16.524487 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.515900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:46868)
0927 10:45:16.516204 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.516219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.516330 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.516695 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.516698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.516930 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.517077 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.517088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.518093 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.518096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.518835 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.518841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.519434 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.519437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.519523 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.520066 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.520081 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.520507 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.522494 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.522498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.522503 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.522809 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.523432 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.523435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.523438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.523532 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.523924 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.523927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.524034 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.524159 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.524279 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
I20260927 10:45:16.524698 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.516014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.10.96.254:32875)
0927 10:45:16.516140 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.516145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.516311 (+ 166us) server_negotiation.cc:407] Connection header received
0927 10:45:16.516416 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.516419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.516479 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.516563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.517229 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.517956 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.518955 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.519320 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.519418 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.520630 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.520633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.520636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.520670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.522366 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.523228 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.523232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.523233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.523330 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.523634 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.523637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.523639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.523842 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.523941 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.524434 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.524547 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:45:16.525518 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.516560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60044)
0927 10:45:16.516809 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.516822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.516915 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.517142 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.517145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.517388 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.517533 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.517541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.518522 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.518526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.519407 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.519417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.520390 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.520394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.520489 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.520974 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.520990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.521390 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.523864 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.523876 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.523877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.524096 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.524370 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.524373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.524375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.524464 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.524964 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.524966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.525040 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.525146 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.525304 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260927 10:45:16.525568 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.516526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.10.96.253:42561)
0927 10:45:16.516628 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.516634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.516906 (+ 272us) server_negotiation.cc:407] Connection header received
0927 10:45:16.516988 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.516991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.517027 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.517111 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.517690 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.518397 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.519563 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.520258 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.520357 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.521533 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.521540 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.521545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.521570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.523773 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.524182 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.524186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.524187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.524273 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.524571 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.524575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.524579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.524755 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.524910 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.525215 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.525370 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":40}
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.254@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
I20260927 10:45:16.538969 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.529408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60048)
0927 10:45:16.529786 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.529841 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.529969 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.530256 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.530262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.530673 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.530791 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.530799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.531492 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.531496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.532479 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.532489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.532901 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.532905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.533008 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.533604 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.533619 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.535175 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.537153 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.537160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.537164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.537490 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.537791 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.537795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.537797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.537885 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.538276 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.538284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.538402 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.538542 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.538720 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":121,"threads_started":1}
I20260927 10:45:16.539600 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.529447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.10.96.253:42561)
0927 10:45:16.529580 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.529584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.529924 (+ 340us) server_negotiation.cc:407] Connection header received
0927 10:45:16.530089 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.530091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.530135 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.530198 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.530923 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.531387 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.532596 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.532799 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.532876 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.535309 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.535313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.535315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.535338 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.537027 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.537596 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.537601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.537603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.537692 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.537981 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.537984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.537985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.538169 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.538276 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.539295 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.539416 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:45:16.541043 16734 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.96.253:42561: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "53234ec893754ca48c2936400df84c38"
member_type: NON_VOTER
last_known_addr {
host: "127.10.96.253"
port: 42561
}
attrs {
promote: true
}
}
dest_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02"
cas_config_opid_index: -1
I20260927 10:45:16.541729 16734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } }
I20260927 10:45:16.543530 17293 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 53234ec893754ca48c2936400df84c38 (127.10.96.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } }
I20260927 10:45:16.544422 17240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:16.545641 17293 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.96.253:42561
WARNING: no policy specified for oryx/127.10.96.252@KRBTEST.COM; defaulting to no policy
I20260927 10:45:16.546120 17198 master.cc:537] Master@127.10.96.253:42561 shutting down...
I20260927 10:45:16.546379 17294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } }
I20260927 10:45:16.546515 17294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:16.546118 17240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Refusing update from remote peer 2231c3f1a56a41c0965b1e7bf3b3ad02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 10:45:16.548583 17294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.252@KRBTEST.COM" created.
I20260927 10:45:16.550972 17198 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:16.551330 17198 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:16.551481 17198 tablet_replica.cc:330] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: stopping tablet replica
I20260927 10:45:16.560724 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.560393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60050)
0927 10:45:16.560575 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.560641 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 10:45:16.561022 16721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.253:42561: connect: Connection refused (error 111)
I20260927 10:45:16.561993 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.561753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60062)
0927 10:45:16.561887 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.561931 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 10:45:16.562203 16721 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 -> Peer 53234ec893754ca48c2936400df84c38 (127.10.96.253:42561): Couldn't send request to peer 53234ec893754ca48c2936400df84c38. Status: Network error: Client connection negotiation failed: client connection to 127.10.96.253:42561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:45:16.566507 17198 master.cc:552] Master@127.10.96.253:42561 shutdown complete.
I20260927 10:45:16.566563 17198 master_runner.cc:305] Clearing existing system tablet
I20260927 10:45:16.566808 17198 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.566870 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.568889 17198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:45:16.569891 17300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.570142 17198 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.570287 17198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:16.570341 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
uuid: "53234ec893754ca48c2936400df84c38"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "b83432d86bbf5b4c5fef0e67590bd756"
server_key_iv: "ce1b0df66d031fc1f5f77c14e0bcf8b6"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.252.keytab.
Entry for principal oryx/127.10.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.252.keytab.
I20260927 10:45:16.572378 17198 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:16.572813 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:39443
--webserver_interface=127.10.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.252.keytab
--principal=oryx/127.10.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:45:16.574963 17198 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:45:16.575058 17198 log.cc:1196] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 10:45:16.575326 17198 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Deleting consensus metadata
I20260927 10:45:16.575508 17198 master_runner.cc:315] Copying system tablet from 127.10.96.254:32875
I20260927 10:45:16.576591 17198 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: tablet copy: Beginning tablet copy session from remote peer at address 127.10.96.254:32875
I20260927 10:45:16.584430 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.577196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.10.96.254:32875)
0927 10:45:16.577343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.577349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.577681 (+ 332us) server_negotiation.cc:407] Connection header received
0927 10:45:16.577830 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.577833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.577883 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.577974 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.578496 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.579046 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.579877 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.580095 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.580203 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.581298 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.581302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.581305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.581332 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.582826 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.583266 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.583269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.583271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.583346 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.583623 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.583626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.583628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.583796 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.583899 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.584153 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.584269 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:45:16.584442 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.577228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:46878)
0927 10:45:16.577587 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.577599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.577724 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.577991 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.577994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.578238 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.578365 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.578374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.579153 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.579160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.579748 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.579764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.580206 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.580209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.580301 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.580836 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.580858 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.581154 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.582958 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.582962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.582964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.583160 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.583438 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.583443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.583445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.583529 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.583889 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.583892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.583992 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.584099 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.584235 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260927 10:45:16.584921 16754 tablet_copy_service.cc:140] P 2231c3f1a56a41c0965b1e7bf3b3ad02: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 53234ec893754ca48c2936400df84c38 ({username='oryx', principal='oryx/127.10.96.253@KRBTEST.COM'} at 127.0.0.1:46878)
I20260927 10:45:16.585103 16754 tablet_copy_service.cc:161] P 2231c3f1a56a41c0965b1e7bf3b3ad02: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 53234ec893754ca48c2936400df84c38 at {username='oryx', principal='oryx/127.10.96.253@KRBTEST.COM'} at 127.0.0.1:46878: session id = 53234ec893754ca48c2936400df84c38-00000000000000000000000000000000
I20260927 10:45:16.586527 16754 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:45:16.588011 17198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:45:16.589798 17198 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: tablet copy: Starting download of 0 data blocks...
I20260927 10:45:16.589988 17198 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: tablet copy: Starting download of 1 WAL segments...
I20260927 10:45:16.591562 17198 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:45:16.593199 16754 tablet_copy_service.cc:342] P 2231c3f1a56a41c0965b1e7bf3b3ad02: Request end of tablet copy session 53234ec893754ca48c2936400df84c38-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.96.253@KRBTEST.COM'} at 127.0.0.1:46878
I20260927 10:45:16.593319 16754 tablet_copy_service.cc:433] P 2231c3f1a56a41c0965b1e7bf3b3ad02: ending tablet copy session 53234ec893754ca48c2936400df84c38-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 53234ec893754ca48c2936400df84c38
I20260927 10:45:16.600072 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:16.601706 17311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.601840 17308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.605890 17309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:16.606323 17198 server_base.cc:1034] running on GCE node
I20260927 10:45:16.606479 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:16.606802 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:16.607937 17198 hybrid_clock.cc:648] HybridClock initialized: now 1790505916607947 us; error 49 us; skew 500 ppm
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:16.609341 17198 init.cc:377] Logged in from keytab as oryx/127.10.96.253@KRBTEST.COM (short username oryx)
I20260927 10:45:16.609851 17198 webserver.cc:460] Webserver started at http://127.10.96.253:46657/ using document root <none> and password file <none>
I20260927 10:45:16.610059 17198 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.610119 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.611172 17198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:45:16.612061 17318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.612278 17198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:45:16.612370 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
uuid: "53234ec893754ca48c2936400df84c38"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "b83432d86bbf5b4c5fef0e67590bd756"
server_key_iv: "ce1b0df66d031fc1f5f77c14e0bcf8b6"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.612453 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.618778 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:16.620885 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:16.625527 17198 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.253:42561
I20260927 10:45:16.625538 17370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.253:42561 every 8 connection(s)
I20260927 10:45:16.626065 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
I20260927 10:45:16.627576 17371 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:45:16.628412 17371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Bootstrap starting.
I20260927 10:45:16.636230 17371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:45:16.636544 17371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: Bootstrap complete.
I20260927 10:45:16.636991 17371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } }
I20260927 10:45:16.637116 17371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 53234ec893754ca48c2936400df84c38, State: Initialized, Role: LEARNER
I20260927 10:45:16.637246 17371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } }
I20260927 10:45:16.637580 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } }
I20260927 10:45:16.637732 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:45:16.637745 17371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:16.641260 17385 catalog_manager.cc:1259] Loaded cluster ID: 77ca7f35a69446c39f57a0db1f742941
I20260927 10:45:16.641325 17385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: loading cluster ID for follower catalog manager: success
I20260927 10:45:16.642800 17385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: acquiring CA information for follower catalog manager: success
I20260927 10:45:16.643839 17385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:45:16.717433 17301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:16.717685 17301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:16.717723 17301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:16.721997 17301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:16.722056 17301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:16.722076 17301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:16.722093 17301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:16.722108 17301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:16.727281 17301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.252.keytab
--principal=oryx/127.10.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:39443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:16.728235 17301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:16.729212 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:16.737255 17390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.737293 17391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.737298 17393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:16.737397 17301 server_base.cc:1034] running on GCE node
I20260927 10:45:16.738090 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:16.739311 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:16.740483 17301 hybrid_clock.cc:648] HybridClock initialized: now 1790505916740468 us; error 28 us; skew 500 ppm
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:16.746210 17301 init.cc:377] Logged in from keytab as oryx/127.10.96.252@KRBTEST.COM (short username oryx)
I20260927 10:45:16.747062 17301 webserver.cc:460] Webserver started at http://127.10.96.252:46049/ using document root <none> and password file <none>
I20260927 10:45:16.747566 17301 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.747622 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.747777 17301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:16.749799 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/instance:
uuid: "f92005e8c3444f3e89f6633ce036c20d"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "f2d3d88fd9e00be2edab9248c3648ebc"
server_key_iv: "b935b5482f5e212ad8c8781a580b2ccb"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.750303 17301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal/instance:
uuid: "f92005e8c3444f3e89f6633ce036c20d"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "f2d3d88fd9e00be2edab9248c3648ebc"
server_key_iv: "b935b5482f5e212ad8c8781a580b2ccb"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.753798 17301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:45:16.755992 17400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.757189 17301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:16.757409 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
uuid: "f92005e8c3444f3e89f6633ce036c20d"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "f2d3d88fd9e00be2edab9248c3648ebc"
server_key_iv: "b935b5482f5e212ad8c8781a580b2ccb"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.757572 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.776629 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:16.779670 17301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:16.779888 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:16.786755 17301 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.252:39443
I20260927 10:45:16.787050 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.252:39443 every 8 connection(s)
I20260927 10:45:16.787721 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
I20260927 10:45:16.790189 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 17301
I20260927 10:45:16.790397 17453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:16.795461 17453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.96.252@KRBTEST.COM
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.252@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:16.805835 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.791656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.10.96.252:39443)
0927 10:45:16.792109 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.792127 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.792167 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:45:16.793147 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.793171 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.793622 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.793999 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.794701 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.796135 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.796984 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.797630 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.797827 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.800702 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.800709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.800720 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.800749 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.802861 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.803324 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.803330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.803334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.803479 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.803783 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.803786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.803788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.804137 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.804350 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.804643 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.804902 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":127,"threads_started":1}
I20260927 10:45:16.806813 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.797302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:46892)
0927 10:45:16.797688 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.797748 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.797901 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.798265 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.798273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.798628 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.798753 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.798759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.799820 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.799825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.800590 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.800598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.801191 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.801196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.801336 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.801853 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.801871 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.803299 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.805262 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.805269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.805274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.805571 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.805858 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.805861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.805864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.805950 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.806388 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.806393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.806476 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.806573 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.806694 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":162,"threads_started":1}
I20260927 10:45:16.806951 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.797432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.10.96.254:32875)
0927 10:45:16.797604 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.797609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.797821 (+ 212us) server_negotiation.cc:407] Connection header received
0927 10:45:16.798057 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.798063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.798123 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.798195 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.798884 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.799661 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.800697 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.801080 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.801159 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.803429 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.803432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.803435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.803460 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.805146 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.805664 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.805667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.805669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.805758 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.806066 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.806070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.806072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.806268 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.806334 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.806693 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.806804 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 10:45:16.807878 17453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.252@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
I20260927 10:45:16.815461 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.808366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60074)
0927 10:45:16.808480 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.808494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.808568 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.808902 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.808904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.809019 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.809096 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.809100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.810138 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.810141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.810671 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.810679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.811103 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.811105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.811172 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.811545 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.811555 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.812710 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.814326 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.814328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.814330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.814502 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.814736 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.814738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.814739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.814804 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.815113 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.815115 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.815183 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.815259 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.815356 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 10:45:16.815558 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.808478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.10.96.253:42561)
0927 10:45:16.808717 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.808720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.808732 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:16.808766 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.808768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.808804 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.808876 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.809249 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.809982 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.810781 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.811004 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.811093 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.812841 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.812844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.812846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.812868 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.814227 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.814582 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.814584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.814586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.814654 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.814880 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.814882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.814883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.815038 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.815110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.815313 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.815404 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260927 10:45:16.816224 17453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.252@KRBTEST.COM for oryx/127.10.96.252@KRBTEST.COM
I20260927 10:45:16.823859 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.816668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60172)
0927 10:45:16.816794 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.816809 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.816881 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.817079 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.817081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.817241 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.817324 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.817329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.818103 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.818106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.818582 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.818588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.819036 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.819040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.819109 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.819568 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.819596 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.820687 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.822410 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.822415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.822418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.822600 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.822887 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.822889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.822891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.822965 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.823321 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.823324 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.823407 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.823528 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.823671 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:45:16.824244 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.816709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.10.96.252:39443)
0927 10:45:16.816826 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.816833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.816847 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:16.816948 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.816953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.816984 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.817072 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.817497 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.817971 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.818689 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.818938 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.819042 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.820815 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.820819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.820821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.820842 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.822300 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.822716 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.822719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.822720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.822791 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.823052 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.823057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.823058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.823217 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.823321 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.824073 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.824152 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:45:16.826917 17453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Bootstrap starting.
I20260927 10:45:16.829414 17453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:16.830116 17453 log.cc:824] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:16.831751 17453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: No bootstrap required, opened a new log
I20260927 10:45:16.834658 17453 raft_consensus.cc:348] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } }
I20260927 10:45:16.834836 17453 raft_consensus.cc:374] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:16.834868 17453 raft_consensus.cc:729] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92005e8c3444f3e89f6633ce036c20d, State: Initialized, Role: FOLLOWER
I20260927 10:45:16.835202 17453 consensus_queue.cc:260] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } }
I20260927 10:45:16.835592 17459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } } }
I20260927 10:45:16.835707 17459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:16.836310 17453 sys_catalog.cc:564] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:45:16.840955 17470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:45:16.841033 17470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:45:16.841650 16925 heartbeater.cc:503] Master 127.10.96.254:32875 was elected leader, sending a full tablet report...
I20260927 10:45:16.843992 17301 master_runner.cc:428] Detected that this master f92005e8c3444f3e89f6633ce036c20d is joining an existing cluster
I20260927 10:45:16.844053 17301 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.96.252:39443
I20260927 10:45:16.854777 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.846187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.10.96.252:39443)
0927 10:45:16.846294 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.846299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.846590 (+ 291us) server_negotiation.cc:407] Connection header received
0927 10:45:16.846733 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.846736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.846772 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.846815 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.847589 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.848380 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.849585 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.849900 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.849969 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.851843 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.851845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.851847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.851867 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.853202 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.853682 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.853684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.853686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.853761 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.854028 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.854029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.854030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.854165 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.854230 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.854493 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.854637 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":41}
I20260927 10:45:16.854802 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.846214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60180)
0927 10:45:16.846509 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.846525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.846641 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.846931 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.846934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.847275 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.847469 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.847477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.848500 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.848503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.849416 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.849427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.850112 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.850115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.850222 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.851265 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.851280 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.851712 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.853307 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.853311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.853313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.853591 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.853849 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.853852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.853854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.853947 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.854250 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.854255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.854346 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.854460 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.854592 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":23,"spinlock_wait_cycles":3456,"thread_start_us":98,"threads_started":1}
I20260927 10:45:16.859294 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16796
I20260927 10:45:16.866338 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:35571
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:32875
--webserver_port=43001
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:45:16.875857 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.846557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60078)
0927 10:45:16.846858 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.846870 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.846986 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.847349 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.847352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.847517 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.847660 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.847667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.848659 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.848661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.849705 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.849715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.850395 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.850399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.850614 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.870295 (+ 19681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.870312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.870987 (+ 675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.874147 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.874150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.874152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.874435 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.874746 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.874749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.874751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.874833 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.875307 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.875310 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.875408 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.875502 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.875657 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":211,"spinlock_wait_cycles":45696,"thread_start_us":74,"threads_started":1}
I20260927 10:45:16.876178 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.846016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.10.96.254:32875)
0927 10:45:16.846148 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.846153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.846552 (+ 399us) server_negotiation.cc:407] Connection header received
0927 10:45:16.846748 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.846751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.846812 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.846866 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.847615 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.848428 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.849840 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.850178 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.850248 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.871204 (+ 20956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.871210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.871216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.871253 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.873836 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.874538 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.874544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.874547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.874646 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.875008 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.875013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.875015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.875277 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.875374 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.875842 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.875977 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:45:16.876390 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.846366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.10.96.253:42561)
0927 10:45:16.846470 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.846476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.846929 (+ 453us) server_negotiation.cc:407] Connection header received
0927 10:45:16.847087 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.847091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.847143 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.847197 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.847789 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.848530 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.849840 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.850178 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.850248 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.871192 (+ 20944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.871197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.871200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.871231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.874014 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.874553 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.874557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.874559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.874642 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.874941 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.874944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.874946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.875166 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.875265 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.876068 (+ 803us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.876245 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:45:16.877604 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.845916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:46898)
0927 10:45:16.846470 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.846486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.846623 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.847015 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.847021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.847286 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.847481 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.847489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.848557 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.848563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.849711 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.849716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.850395 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.850399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.850595 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.870178 (+ 19583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.870204 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.870976 (+ 772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.873989 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.873995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.874000 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.874332 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.874781 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.874786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.874791 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.874889 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.875458 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.875463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.875627 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.877085 (+ 1458us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.877276 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":228,"spinlock_wait_cycles":1280,"thread_start_us":99,"threads_started":1}
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505915, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.254@KRBTEST.COM for oryx/127.10.96.252@KRBTEST.COM
I20260927 10:45:16.890691 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.881379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60192)
0927 10:45:16.881592 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.881606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.881677 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.882113 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.882116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.882296 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.882404 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.882409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.883437 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.883440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.884305 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.884314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.885041 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.885047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.885170 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.885709 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.885723 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.887175 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.889089 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.889094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.889098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.889340 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.889664 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.889668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.889670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.889753 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.890180 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.890183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.890275 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.890384 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.890523 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 10:45:16.891218 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.881443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.10.96.252:39443)
0927 10:45:16.881815 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.881819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.881831 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:16.881893 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.881896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.881942 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.882005 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.882544 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.883313 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.884439 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.884802 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.884912 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.887328 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.887332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.887335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.887357 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.888971 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.889445 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.889451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.889453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.889563 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.889844 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.889849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.889851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.890046 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.890130 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.890950 (+ 820us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.891055 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304}
I20260927 10:45:16.892176 16734 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.96.252:39443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d"
member_type: NON_VOTER
last_known_addr {
host: "127.10.96.252"
port: 39443
}
attrs {
promote: true
}
}
dest_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02"
cas_config_opid_index: 5
I20260927 10:45:16.892788 16734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } }
I20260927 10:45:16.893821 17295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f92005e8c3444f3e89f6633ce036c20d (127.10.96.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.894848 17427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:16.895227 17295 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.96.252:39443
I20260927 10:45:16.895396 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.895483 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:16.896138 17301 master.cc:537] Master@127.10.96.252:39443 shutting down...
I20260927 10:45:16.896243 17427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Refusing update from remote peer 2231c3f1a56a41c0965b1e7bf3b3ad02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:45:16.896848 17294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:45:16.899266 16720 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 -> Peer f92005e8c3444f3e89f6633ce036c20d (127.10.96.252:39443): Couldn't send request to peer f92005e8c3444f3e89f6633ce036c20d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:45:16.901542 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.893989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60088)
0927 10:45:16.896972 (+ 2983us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.896986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.897073 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.897507 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.897510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.897526 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:16.897605 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.897611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.898662 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.898665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.899925 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.899937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.901113 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.901119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.901204 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.901254 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.901335 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.901406 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2923}
I20260927 10:45:16.901937 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.894206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.10.96.253:42561)
0927 10:45:16.896892 (+ 2686us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.896896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.897153 (+ 257us) server_negotiation.cc:407] Connection header received
0927 10:45:16.897221 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.897224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.897279 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.897367 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:16.897721 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.898513 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.900066 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.900807 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.900907 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.901051 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.901732 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.901811 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2594}
I20260927 10:45:16.902344 17345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 LEARNER]: Refusing update from remote peer 2231c3f1a56a41c0965b1e7bf3b3ad02: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 10:45:16.902729 17294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 10:45:16.903676 17301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:16.903939 17301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:16.904031 17301 tablet_replica.cc:330] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: stopping tablet replica
I20260927 10:45:16.904626 17345 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f92005e8c3444f3e89f6633ce036c20d (127.10.96.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.906944 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2231c3f1a56a41c0965b1e7bf3b3ad02. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.907065 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:45:16.907152 17374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: NON_VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: true } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.907210 17374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: LEARNER
I20260927 10:45:16.907779 17294 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: attempting to promote NON_VOTER 53234ec893754ca48c2936400df84c38 to VOTER
I20260927 10:45:16.908183 17294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } }
I20260927 10:45:16.909303 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.909054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60200)
0927 10:45:16.909187 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.909238 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:39443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 10:45:16.909332 17345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 LEARNER]: Refusing update from remote peer 2231c3f1a56a41c0965b1e7bf3b3ad02: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 10:45:16.909822 17293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 10:45:16.910181 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.909789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60212)
0927 10:45:16.910074 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.910119 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:39443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231}
W20260927 10:45:16.910291 16720 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 -> Peer f92005e8c3444f3e89f6633ce036c20d (127.10.96.252:39443): Couldn't send request to peer f92005e8c3444f3e89f6633ce036c20d. Status: Network error: Client connection negotiation failed: client connection to 127.10.96.252:39443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:45:16.911432 17295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53234ec893754ca48c2936400df84c38 (127.10.96.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.913241 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.913184 17344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53234ec893754ca48c2936400df84c38 (127.10.96.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.913357 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:16.913440 17294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.913528 17294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:16.914641 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.915042 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:16.919801 17301 master.cc:552] Master@127.10.96.252:39443 shutdown complete.
I20260927 10:45:16.919854 17301 master_runner.cc:305] Clearing existing system tablet
I20260927 10:45:16.920094 17301 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.920158 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.921490 17301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 10:45:16.922361 17483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.922598 17301 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.922719 17301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:16.922752 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
uuid: "f92005e8c3444f3e89f6633ce036c20d"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "f2d3d88fd9e00be2edab9248c3648ebc"
server_key_iv: "b935b5482f5e212ad8c8781a580b2ccb"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.924657 17301 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:16.926961 17301 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 10:45:16.927045 17301 log.cc:1196] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 10:45:16.927273 17301 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Deleting consensus metadata
I20260927 10:45:16.927451 17301 master_runner.cc:315] Copying system tablet from 127.10.96.254:32875
I20260927 10:45:16.928531 17301 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: tablet copy: Beginning tablet copy session from remote peer at address 127.10.96.254:32875
I20260927 10:45:16.938125 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.929042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:46912)
0927 10:45:16.929368 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:16.929383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:16.929524 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:16.929787 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:16.929790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:16.930002 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:16.930150 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.930158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.931198 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.931201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.932129 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:16.932140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.932728 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.932732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:16.932830 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.933789 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:16.933804 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:16.934204 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:16.936271 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.936274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.936277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.936529 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.936862 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:16.936865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:16.936867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:16.936962 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:16.937479 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:16.937482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:16.937584 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 10:45:16.937691 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 10:45:16.937822 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260927 10:45:16.938140 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:16.929131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.10.96.254:32875)
0927 10:45:16.929249 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:16.929253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:16.929496 (+ 243us) server_negotiation.cc:407] Connection header received
0927 10:45:16.929606 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:16.929609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:16.929662 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:16.929756 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:16.930272 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.931081 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.932254 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:16.932606 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:16.932712 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:16.934360 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:16.934364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:16.934367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:16.934394 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:16.936151 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.936632 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.936635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.936637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.936728 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:16.937066 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:16.937073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:16.937076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:16.937358 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:16.937476 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:16.937758 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 10:45:16.937937 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:45:16.938902 16754 tablet_copy_service.cc:140] P 2231c3f1a56a41c0965b1e7bf3b3ad02: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f92005e8c3444f3e89f6633ce036c20d ({username='oryx', principal='oryx/127.10.96.252@KRBTEST.COM'} at 127.0.0.1:46912)
I20260927 10:45:16.939010 16754 tablet_copy_service.cc:161] P 2231c3f1a56a41c0965b1e7bf3b3ad02: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f92005e8c3444f3e89f6633ce036c20d at {username='oryx', principal='oryx/127.10.96.252@KRBTEST.COM'} at 127.0.0.1:46912: session id = f92005e8c3444f3e89f6633ce036c20d-00000000000000000000000000000000
I20260927 10:45:16.940047 16754 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:45:16.941313 17301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 10:45:16.943555 17301 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: tablet copy: Starting download of 0 data blocks...
I20260927 10:45:16.943769 17301 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: tablet copy: Starting download of 1 WAL segments...
I20260927 10:45:16.945223 17301 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:45:16.946918 16754 tablet_copy_service.cc:342] P 2231c3f1a56a41c0965b1e7bf3b3ad02: Request end of tablet copy session f92005e8c3444f3e89f6633ce036c20d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.96.252@KRBTEST.COM'} at 127.0.0.1:46912
I20260927 10:45:16.946992 16754 tablet_copy_service.cc:433] P 2231c3f1a56a41c0965b1e7bf3b3ad02: ending tablet copy session f92005e8c3444f3e89f6633ce036c20d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f92005e8c3444f3e89f6633ce036c20d
I20260927 10:45:16.954653 17301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:16.956430 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:16.958365 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:16.958595 17301 server_base.cc:1034] running on GCE node
W20260927 10:45:16.958770 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:16.959014 17301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:16.959299 17301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:16.960443 17301 hybrid_clock.cc:648] HybridClock initialized: now 1790505916960430 us; error 32 us; skew 500 ppm
Sep 27 10:45:16 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505916, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:16.963101 17301 init.cc:377] Logged in from keytab as oryx/127.10.96.252@KRBTEST.COM (short username oryx)
I20260927 10:45:16.963594 17301 webserver.cc:460] Webserver started at http://127.10.96.252:32903/ using document root <none> and password file <none>
I20260927 10:45:16.963794 17301 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:16.963837 17301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:16.965234 17301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:16.966305 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:16.966548 17301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 10:45:16.966617 17301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
uuid: "f92005e8c3444f3e89f6633ce036c20d"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "f2d3d88fd9e00be2edab9248c3648ebc"
server_key_iv: "b935b5482f5e212ad8c8781a580b2ccb"
server_key_version: "encryptionkey@0"
I20260927 10:45:16.966687 17301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:16.973767 17301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:16.976130 17301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:16.981354 17301 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.252:39443
I20260927 10:45:16.981410 17552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.252:39443 every 8 connection(s)
I20260927 10:45:16.982086 17301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
I20260927 10:45:16.983950 17553 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:45:16.985018 17553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Bootstrap starting.
I20260927 10:45:16.995832 17553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:45:16.996106 17553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Bootstrap complete.
I20260927 10:45:16.996634 17553 raft_consensus.cc:348] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } }
I20260927 10:45:16.996766 17553 raft_consensus.cc:729] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f92005e8c3444f3e89f6633ce036c20d, State: Initialized, Role: LEARNER
I20260927 10:45:16.996855 17553 consensus_queue.cc:260] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } }
I20260927 10:45:16.997145 17556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:16.997278 17553 sys_catalog.cc:564] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:16.997362 17556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: This master's current role is: LEARNER
I20260927 10:45:17.001607 17567 catalog_manager.cc:1259] Loaded cluster ID: 77ca7f35a69446c39f57a0db1f742941
I20260927 10:45:17.001660 17567 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: loading cluster ID for follower catalog manager: success
I20260927 10:45:17.003801 17567 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: acquiring CA information for follower catalog manager: success
I20260927 10:45:17.005128 17567 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 10:45:17.013315 17479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:17.013619 17479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:17.013706 17479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:17.017830 17479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:17.017915 17479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:17.018010 17479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:17.023387 17479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:35571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=43001
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:17.024394 17479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:17.025383 17479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:17.035256 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:17.035234 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:17.035269 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:17.035382 17479 server_base.cc:1034] running on GCE node
I20260927 10:45:17.035822 17479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:17.036383 17479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:17.037554 17479 hybrid_clock.cc:648] HybridClock initialized: now 1790505917037526 us; error 42 us; skew 500 ppm
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:17.040351 17479 init.cc:377] Logged in from keytab as oryx/127.10.96.193@KRBTEST.COM (short username oryx)
I20260927 10:45:17.041141 17479 webserver.cc:460] Webserver started at http://127.10.96.193:43001/ using document root <none> and password file <none>
I20260927 10:45:17.041747 17479 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:17.041864 17479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:17.045636 17479 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 10:45:17.047693 17059 heartbeater.cc:503] Master 127.10.96.254:32875 was elected leader, sending a full tablet report...
I20260927 10:45:17.047899 17582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.049400 17479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:45:17.049561 17479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "d618480d3ce74fd1b605fa2c691ff70c"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "4a95ac106efcd9fad085ee248c0e7cc1"
server_key_iv: "3285354c03e6a2e8b2f41c3fd2d3d517"
server_key_version: "encryptionkey@0"
I20260927 10:45:17.050124 17479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:17.063186 17479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:17.066051 17479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:17.066256 17479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:17.066877 17479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:17.067817 17479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:17.067865 17479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.067931 17479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:17.067961 17479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.077798 17479 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:35571
I20260927 10:45:17.077836 17695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:35571 every 8 connection(s)
I20260927 10:45:17.078807 17479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:17.082844 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 17479
I20260927 10:45:17.083033 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16930
I20260927 10:45:17.091936 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:44993
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:32875
--webserver_port=42995
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.252@KRBTEST.COM
I20260927 10:45:17.099115 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.081611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:57995 (local address 127.10.96.252:39443)
0927 10:45:17.082330 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.082335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.084576 (+ 2241us) server_negotiation.cc:407] Connection header received
0927 10:45:17.084758 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.084764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.084826 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.084893 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.086361 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.087430 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089318 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.089690 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089769 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.094443 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.094448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.094450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.094483 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.096387 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097120 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.097124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.097126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.097243 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097740 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.097745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.097750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.097989 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.098069 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.098817 (+ 748us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.098945 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":418,"threads_started":1}
I20260927 10:45:17.099112 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.082680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:57169 (local address 127.10.96.254:32875)
0927 10:45:17.082801 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.082807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.083578 (+ 771us) server_negotiation.cc:407] Connection header received
0927 10:45:17.084194 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.084197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.084258 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.084310 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.086361 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.087172 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089319 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.089684 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089766 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.094171 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.094177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.094181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.094219 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.096112 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097471 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.097495 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.097496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.097579 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097898 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.097901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.097903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.098112 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.098177 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.098817 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.098944 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:45:17.099647 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.081713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:38663 (local address 127.10.96.253:42561)
0927 10:45:17.081990 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.082016 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.082995 (+ 979us) server_negotiation.cc:407] Connection header received
0927 10:45:17.084192 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.084195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.084254 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.084310 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.086351 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.087261 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089316 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.089768 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089862 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.094538 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.094542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.094544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.094576 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.097203 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097973 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.097980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.097984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.098089 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.098537 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.098543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.098546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.098821 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.098921 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.099348 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.099462 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260927 10:45:17.100939 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.081241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.10.96.193:38663)
0927 10:45:17.082770 (+ 1529us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.082866 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.083950 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.084706 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.084714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.085480 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.086205 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.086218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.087419 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.087425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.089151 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.089164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.090047 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.090054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.090282 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.091215 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.091235 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.094398 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.097325 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.097336 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.097779 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.098219 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.098224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.098229 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.098390 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.098986 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.098992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.099103 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.099919 (+ 816us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.100090 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"mutex_wait_us":692,"spinlock_wait_cycles":2304,"thread_start_us":105,"threads_started":1}
I20260927 10:45:17.100937 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.081242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.10.96.193:57995)
0927 10:45:17.084495 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.084512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.084643 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.085070 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.085076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.085886 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.086185 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.086204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.087560 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.087563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.089155 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.089165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.089966 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.090269 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.091145 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.091156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.094282 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.096750 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.096753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.096755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.097000 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.097353 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.097357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.097616 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.098137 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.098146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.098516 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.099386 (+ 870us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.099768 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":747,"spinlock_wait_cycles":8064,"thread_start_us":182,"threads_started":1}
I20260927 10:45:17.101207 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.082345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:57169)
0927 10:45:17.083485 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.083501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.083950 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.084706 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.084714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.085354 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.086184 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.086204 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.087352 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.087355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.089155 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.089165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.089966 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.089970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.090302 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.091114 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.091134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.093983 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.096266 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.096272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.096291 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.096650 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.097699 (+ 1049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.097704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.097709 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.097793 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.098252 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.098255 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.098516 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.099283 (+ 767us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.099720 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":419,"spinlock_wait_cycles":8576,"thread_start_us":77,"threads_started":1}
I20260927 10:45:17.102345 17698 heartbeater.cc:347] Connected to a master server at 127.10.96.253:42561
I20260927 10:45:17.102401 17697 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32875
I20260927 10:45:17.102589 17698 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.102588 17697 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.103067 17696 heartbeater.cc:347] Connected to a master server at 127.10.96.252:39443
I20260927 10:45:17.103145 17696 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.103233 17697 heartbeater.cc:511] Master 127.10.96.254:32875 requested a full tablet report, sending...
I20260927 10:45:17.103288 17696 heartbeater.cc:511] Master 127.10.96.252:39443 requested a full tablet report, sending...
I20260927 10:45:17.103804 17698 heartbeater.cc:511] Master 127.10.96.253:42561 requested a full tablet report, sending...
I20260927 10:45:17.104597 16734 ts_manager.cc:194] Re-registered known tserver with Master: d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571)
I20260927 10:45:17.105074 17509 ts_manager.cc:194] Registered new tserver with Master: d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571)
I20260927 10:45:17.105340 16734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:57169
I20260927 10:45:17.106197 17327 ts_manager.cc:194] Registered new tserver with Master: d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571)
W20260927 10:45:17.207620 17704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:17.207895 17704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:17.207970 17704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:17.211943 17704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:17.212023 17704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:17.212105 17704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:17.217278 17704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:44993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=42995
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:17.218281 17704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:17.219273 17704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:17.228766 17710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:17.228763 17712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:17.228853 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:17.228942 17704 server_base.cc:1034] running on GCE node
I20260927 10:45:17.229307 17704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:17.229952 17704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:17.231139 17704 hybrid_clock.cc:648] HybridClock initialized: now 1790505917231119 us; error 28 us; skew 500 ppm
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:17.233817 17704 init.cc:377] Logged in from keytab as oryx/127.10.96.194@KRBTEST.COM (short username oryx)
I20260927 10:45:17.234694 17704 webserver.cc:460] Webserver started at http://127.10.96.194:42995/ using document root <none> and password file <none>
I20260927 10:45:17.235235 17704 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:17.235345 17704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:17.239604 17704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:45:17.241667 17719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.242940 17704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:45:17.243036 17704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "1dbbf06c99e94029b60d7f9a26fda666"
format_stamp: "Formatted at 2026-09-27 10:45:15 on dist-test-slave-wmg0"
server_key: "1394aecd7d2edb303ea437145e21b18b"
server_key_iv: "d766537064a0fd2765329517110c24af"
server_key_version: "encryptionkey@0"
I20260927 10:45:17.243566 17704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:17.254954 17704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:17.258152 17704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:17.258307 17704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:17.258844 17704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:17.259675 17704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:17.259720 17704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.259761 17704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:17.259778 17704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.265606 17193 heartbeater.cc:503] Master 127.10.96.254:32875 was elected leader, sending a full tablet report...
I20260927 10:45:17.271076 17704 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:44993
I20260927 10:45:17.271106 17832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:44993 every 8 connection(s)
I20260927 10:45:17.272413 17704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:17.279587 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 17704
I20260927 10:45:17.279783 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 17064
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:17.287745 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.275676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:45707 (local address 127.10.96.253:42561)
0927 10:45:17.275802 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.275806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.276152 (+ 346us) server_negotiation.cc:407] Connection header received
0927 10:45:17.276510 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.276513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.276562 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.276618 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.278139 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.278716 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.280031 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.280382 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.280469 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.283339 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.283346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.283352 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.283387 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.285242 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.285916 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.285921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.285922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.286011 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.286395 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.286399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.286400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.286632 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.286739 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.287353 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.287494 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:45:17.287753 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.274296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:47131 (local address 127.10.96.254:32875)
0927 10:45:17.274491 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.274498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.276012 (+ 1514us) server_negotiation.cc:407] Connection header received
0927 10:45:17.276744 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.276749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.276801 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.276882 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.278152 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.278959 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.280319 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.280725 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.280814 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.283973 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.283977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.283979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.284006 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.285917 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.286375 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.286379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.286380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.286470 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.286767 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.286770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.286772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.287011 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.287086 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.287465 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.287589 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 10:45:17.288713 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:38787
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:32875
--webserver_port=38243
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.252@KRBTEST.COM
I20260927 10:45:17.289505 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.274694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:47131)
0927 10:45:17.275872 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.275885 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.276255 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.276919 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.276923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.277327 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.277990 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.278005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.279104 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.279107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.280121 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.280134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.280980 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.280986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.281124 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.282400 (+ 1276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.282418 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.283839 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.286024 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.286027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.286031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.286251 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.286572 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.286575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.286576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.286655 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.287165 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.287169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.287261 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.287962 (+ 701us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.288369 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":218,"spinlock_wait_cycles":17152,"thread_start_us":132,"threads_started":1}
I20260927 10:45:17.290887 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.275568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.10.96.194:45707)
0927 10:45:17.276085 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.276096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.276309 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.276791 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.276794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.277303 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.277988 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.278002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.278859 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.278862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.279877 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.279889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.280675 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.280681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.280878 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.281519 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.281546 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.283149 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.285364 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.285370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.285386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.285786 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.286119 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.286121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.286123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.286280 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.286745 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.286751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.287056 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.287959 (+ 903us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.290328 (+ 2369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":2092,"spinlock_wait_cycles":26240,"thread_start_us":80,"threads_started":1}
I20260927 10:45:17.291004 17834 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32875
I20260927 10:45:17.291245 17834 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.291574 17835 heartbeater.cc:347] Connected to a master server at 127.10.96.253:42561
I20260927 10:45:17.291667 17835 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.291875 17835 heartbeater.cc:511] Master 127.10.96.253:42561 requested a full tablet report, sending...
I20260927 10:45:17.291873 17834 heartbeater.cc:511] Master 127.10.96.254:32875 requested a full tablet report, sending...
I20260927 10:45:17.293012 16734 ts_manager.cc:194] Re-registered known tserver with Master: 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993)
I20260927 10:45:17.293103 17327 ts_manager.cc:194] Registered new tserver with Master: 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993)
I20260927 10:45:17.293787 16734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:47131
I20260927 10:45:17.294102 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.274306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:42673 (local address 127.10.96.252:39443)
0927 10:45:17.274592 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.274596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.275321 (+ 725us) server_negotiation.cc:407] Connection header received
0927 10:45:17.276475 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.276481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.276540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.276623 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.278153 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.278959 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.286391 (+ 7432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.286774 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.286882 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.289913 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.289916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.289919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.289950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.291852 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.292442 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.292448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.292453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.292543 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.292853 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.292859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.292863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.293196 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.293302 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.293702 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.293899 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260927 10:45:17.294828 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.274694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.10.96.194:42673)
0927 10:45:17.275156 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.275208 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.276255 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.276772 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.276778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.277327 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.277990 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.278005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.279104 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.279107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.282203 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.282216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.286892 (+ 4676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.286896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.286987 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.287479 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.287491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.289538 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.292023 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.292026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.292029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.292312 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.292655 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.292660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.292665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.292751 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.293361 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.293366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.293512 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.294379 (+ 867us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.294553 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":2101,"spinlock_wait_cycles":18560,"thread_start_us":103,"threads_started":1}
I20260927 10:45:17.295912 17833 heartbeater.cc:347] Connected to a master server at 127.10.96.252:39443
I20260927 10:45:17.296012 17833 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.296181 17833 heartbeater.cc:511] Master 127.10.96.252:39443 requested a full tablet report, sending...
I20260927 10:45:17.296983 17509 ts_manager.cc:194] Registered new tserver with Master: 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993)
I20260927 10:45:17.310484 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.306770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60224)
0927 10:45:17.306914 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.306930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.307018 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.307272 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.307275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.307290 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:17.307386 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.307392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.308072 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.308076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.309307 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.309317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.310008 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.310013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.310116 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.310141 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.310260 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.310340 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:45:17.310518 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.306877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.10.96.252:39443)
0927 10:45:17.306994 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.306998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.307010 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:17.307112 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.307114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.307160 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.307238 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:17.307482 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.307967 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.309427 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.309896 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.309981 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.310094 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.310293 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.310357 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:45:17.311251 17527 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 10:45:17.311619 17556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2231c3f1a56a41c0965b1e7bf3b3ad02. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:17.311753 17556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: This master's current role is: LEARNER
W20260927 10:45:17.404764 17840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:17.405066 17840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:17.405140 17840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:17.409125 17840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:17.409209 17840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:17.409310 17840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:17.414665 17840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:38787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=38243
--tserver_master_addrs=127.10.96.254:32875,127.10.96.253:42561,127.10.96.252:39443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:17.415697 17840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:17.416699 17840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:17.426275 17845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:17.426275 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:17.426483 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:17.426353 17840 server_base.cc:1034] running on GCE node
I20260927 10:45:17.426893 17840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:17.427474 17840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:17.428666 17840 hybrid_clock.cc:648] HybridClock initialized: now 1790505917428648 us; error 28 us; skew 500 ppm
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:17.431398 17840 init.cc:377] Logged in from keytab as oryx/127.10.96.195@KRBTEST.COM (short username oryx)
I20260927 10:45:17.432363 17840 webserver.cc:460] Webserver started at http://127.10.96.195:38243/ using document root <none> and password file <none>
I20260927 10:45:17.432934 17840 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:17.433043 17840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:17.436800 17840 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:45:17.439015 17855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.440333 17840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:45:17.440459 17840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "dfba3370ab7e43be9d07571f4d24e5cd"
format_stamp: "Formatted at 2026-09-27 10:45:16 on dist-test-slave-wmg0"
server_key: "35320606e377eb6f277f7d43faae8313"
server_key_iv: "49b5896f1a319a4a0f150e34913b70aa"
server_key_version: "encryptionkey@0"
I20260927 10:45:17.440994 17840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:17.450992 17840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:17.454180 17840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:17.454376 17840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:17.454965 17840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:17.455788 17840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:17.455834 17840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.455871 17840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:17.455917 17840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:17.464447 17840 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:38787
I20260927 10:45:17.464470 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:38787 every 8 connection(s)
I20260927 10:45:17.465435 17840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:17.466887 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 17840
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.252@KRBTEST.COM
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.253@KRBTEST.COM
I20260927 10:45:17.481068 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.468679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.10.96.254:32875)
0927 10:45:17.470236 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.470240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.470252 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:17.470288 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.470294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.470340 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.471326 (+ 986us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.471415 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.472237 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.473495 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.473844 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.474355 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.476583 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.476587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.476590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.476615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.478609 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.479236 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.479241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.479244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.479337 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.479677 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.479681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.479683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.479912 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.480050 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.480728 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.480872 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":65,"threads_started":1}
I20260927 10:45:17.482486 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.467405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:43493 (local address 127.10.96.252:39443)
0927 10:45:17.468629 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.468634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.469795 (+ 1161us) server_negotiation.cc:407] Connection header received
0927 10:45:17.470093 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.470097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.470155 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.470208 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.471923 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.472650 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.474598 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.474941 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.475013 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.478442 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.478449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.478451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.478474 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.480152 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.480764 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.480768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.480769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.480851 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.481245 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.481260 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.481261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.481478 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.481590 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.482167 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.482288 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132}
Sep 27 10:45:17 dist-test-slave-wmg0 krb5kdc[16689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505917, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:17.484113 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.469129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.10.96.195:43493)
0927 10:45:17.469708 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.469722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.469871 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.470528 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.470538 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.471071 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.471772 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.471790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.472810 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.472813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.474469 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.474485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.475200 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.475204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.475427 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.476102 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.476125 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.478265 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.480281 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.480290 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.480308 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.480657 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.480957 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.480960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.480962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.481127 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.481606 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.481613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.481901 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.482682 (+ 781us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.483041 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":99,"spinlock_wait_cycles":1792,"thread_start_us":117,"threads_started":1}
I20260927 10:45:17.485731 17969 heartbeater.cc:347] Connected to a master server at 127.10.96.252:39443
I20260927 10:45:17.485986 17969 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.486104 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.469828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:54137 (local address 127.10.96.253:42561)
0927 10:45:17.470864 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.470868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.470881 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:17.470920 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.470923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.470970 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.474256 (+ 3286us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.474372 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.475169 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.477498 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.477899 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.478010 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.481246 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.481250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.481252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.481287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.483115 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.483620 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.483625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.483628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.483709 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.483988 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.483994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.483998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.484224 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.484660 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.485830 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.485948 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":73,"threads_started":1}
I20260927 10:45:17.486654 17969 heartbeater.cc:511] Master 127.10.96.252:39443 requested a full tablet report, sending...
I20260927 10:45:17.483995 16734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:45:17.488444 17509 ts_manager.cc:194] Registered new tserver with Master: dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787)
I20260927 10:45:17.488669 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.468868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.10.96.252:39443)
0927 10:45:17.470084 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.470095 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.470611 (+ 516us) server_negotiation.cc:407] Connection header received
0927 10:45:17.470653 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.470656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.470705 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.470788 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.471692 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.472499 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.475469 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.475803 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.475881 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.479598 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.479604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.479608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.479637 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.481847 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.484014 (+ 2167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.484017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.484019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.484087 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.484624 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.484629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.484631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.484846 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.484956 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.488340 (+ 3384us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.488454 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":83,"threads_started":1}
W20260927 10:45:17.488996 16734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:17.486665 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.469487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.10.96.195:54137)
0927 10:45:17.470071 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.470083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.470162 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.473172 (+ 3010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.473176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.473349 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.473501 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.473508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.476391 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.476395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.477326 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.477336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.478135 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.478139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.478231 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.479125 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.479144 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.481086 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.483228 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.483231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.483233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.483503 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.483810 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.483814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.483816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.483891 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.484824 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.484827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.484913 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.485008 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.485134 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":72,"threads_started":1}
I20260927 10:45:17.490131 17971 heartbeater.cc:347] Connected to a master server at 127.10.96.253:42561
I20260927 10:45:17.490171 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.469447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.10.96.253:42561)
0927 10:45:17.469591 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.469595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.473013 (+ 3418us) server_negotiation.cc:407] Connection header received
0927 10:45:17.473187 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.473190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.473234 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.473285 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.474279 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.475335 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.477944 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.478264 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.478351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.486521 (+ 8170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.486525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.486540 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.486563 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.488365 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.488813 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.488817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.488818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.488895 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.489207 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.489210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.489212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.489431 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.489579 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.489902 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.490011 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260927 10:45:17.490221 17971 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.490381 17971 heartbeater.cc:511] Master 127.10.96.253:42561 requested a full tablet report, sending...
I20260927 10:45:17.491055 17329 ts_manager.cc:194] Registered new tserver with Master: dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787)
I20260927 10:45:17.491499 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.467347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:42781 (local address 127.10.96.254:32875)
0927 10:45:17.467470 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.467474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.468857 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 10:45:17.470103 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.470110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.470167 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.470230 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.471919 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.472642 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.475509 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.475836 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.475898 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.485267 (+ 9369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.485272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.485277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.485306 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.489701 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.490171 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.490175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.490176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.490263 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.490603 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.490610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.490614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.490828 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.490921 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.491243 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.491381 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
I20260927 10:45:17.491874 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.467820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:42781)
0927 10:45:17.468703 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.468752 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.469864 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.470527 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.470536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.471214 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:17.471773 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.471790 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.472808 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.472811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.475393 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.475403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.476027 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.476030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.476120 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.479809 (+ 3689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:17.479827 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:17.483880 (+ 4053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:17.489830 (+ 5950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.489833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.489835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.490065 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.490384 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:17.490388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:17.490391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:17.490478 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:17.490965 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:17.490968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:17.491050 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.491638 (+ 588us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.491760 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":1535,"spinlock_wait_cycles":6016,"thread_start_us":543,"threads_started":1}
I20260927 10:45:17.492411 17970 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32875
I20260927 10:45:17.492489 17970 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:17.492681 17970 heartbeater.cc:511] Master 127.10.96.254:32875 requested a full tablet report, sending...
I20260927 10:45:17.493094 17988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: NON_VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: true } } }
I20260927 10:45:17.493220 17988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:17.493709 16733 ts_manager.cc:194] Re-registered known tserver with Master: dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787)
I20260927 10:45:17.494359 16733 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:42781
I20260927 10:45:17.496057 17295 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: attempting to promote NON_VOTER f92005e8c3444f3e89f6633ce036c20d to VOTER
I20260927 10:45:17.496374 17295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:17.497562 17526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 LEARNER]: Refusing update from remote peer 2231c3f1a56a41c0965b1e7bf3b3ad02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:45:17.497952 17988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 10:45:17.499106 17344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Refusing update from remote peer 2231c3f1a56a41c0965b1e7bf3b3ad02: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 10:45:17.499462 17988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 10:45:17.500294 17295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f92005e8c3444f3e89f6633ce036c20d (127.10.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.500865 17526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f92005e8c3444f3e89f6633ce036c20d (127.10.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.501797 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.501894 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:17.501834 17344 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f92005e8c3444f3e89f6633ce036c20d (127.10.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.502044 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.502111 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:17.502218 17295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.502285 17295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:17.502849 17556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.502946 17556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:17.503854 17996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:17.503997 17996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:17.511346 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.505630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:44993 (local address 127.0.0.1:52840)
0927 10:45:17.505764 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.505781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.505853 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.507282 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.507285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.507301 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:17.507395 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.507402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.509206 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.509212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.510032 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.510040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.510986 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.510989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.511078 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.511102 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.511152 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.511216 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":2}
I20260927 10:45:17.511988 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.505862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:38787 (local address 127.0.0.1:51818)
0927 10:45:17.506105 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.506117 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.506185 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.507625 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.507628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.507645 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:17.507784 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.507794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.509154 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.509157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.510443 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.510454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.511528 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.511535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.511632 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.511656 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.511775 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.511846 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":109,"spinlock_wait_cycles":2944,"thread_start_us":106,"threads_started":1}
I20260927 10:45:17.512100 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.506628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.10.96.194:44993)
0927 10:45:17.506888 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.506897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.506916 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:45:17.506979 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.506983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.507146 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.507311 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:17.507489 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.509117 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.510150 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.510854 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.511691 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.511828 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.511875 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.511959 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260927 10:45:17.512454 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.506911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.10.96.195:38787)
0927 10:45:17.507172 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.507183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.507204 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:45:17.507286 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.507293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.507484 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.507649 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:17.507914 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.509049 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.510564 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.511374 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.511483 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.511636 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.511809 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.511903 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
I20260927 10:45:17.512974 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.506186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35571 (local address 127.0.0.1:38610)
0927 10:45:17.506388 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.506498 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.506588 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.507663 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.507665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.507692 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:17.507828 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.507837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.509032 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.509037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.510448 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.510454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.512622 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.512626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.512708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.512730 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.512777 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.512842 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":87,"spinlock_wait_cycles":384,"thread_start_us":90,"threads_started":1}
I20260927 10:45:17.513661 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.506384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.10.96.193:35571)
0927 10:45:17.507287 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.507297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.507316 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:45:17.507376 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.507380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.507519 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.507978 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:17.508123 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.508913 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.511697 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.512489 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.513223 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.513374 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.513422 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.513528 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":133,"threads_started":1}
I20260927 10:45:17.515058 17903 tablet_service.cc:1467] Processing CreateTablet for tablet b2d35d9ea3074119a07945f441b71e2a (DEFAULT_TABLE table=test-table [id=a35175dcdfda4321a392aacc59c4ca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:17.515120 17767 tablet_service.cc:1467] Processing CreateTablet for tablet b2d35d9ea3074119a07945f441b71e2a (DEFAULT_TABLE table=test-table [id=a35175dcdfda4321a392aacc59c4ca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:17.516040 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d35d9ea3074119a07945f441b71e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:17.516109 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d35d9ea3074119a07945f441b71e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:17.516010 17630 tablet_service.cc:1467] Processing CreateTablet for tablet b2d35d9ea3074119a07945f441b71e2a (DEFAULT_TABLE table=test-table [id=a35175dcdfda4321a392aacc59c4ca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:17.516865 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d35d9ea3074119a07945f441b71e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:17.522955 18002 tablet_bootstrap.cc:492] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Bootstrap starting.
I20260927 10:45:17.524206 18003 tablet_bootstrap.cc:492] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Bootstrap starting.
I20260927 10:45:17.524910 18004 tablet_bootstrap.cc:492] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Bootstrap starting.
I20260927 10:45:17.525139 18002 tablet_bootstrap.cc:654] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:17.526031 18002 log.cc:824] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:17.526531 18003 tablet_bootstrap.cc:654] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:17.526767 18004 tablet_bootstrap.cc:654] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:17.527311 18003 log.cc:824] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:17.527431 18004 log.cc:824] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:17.528154 18002 tablet_bootstrap.cc:492] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: No bootstrap required, opened a new log
I20260927 10:45:17.528330 18002 ts_tablet_manager.cc:1397] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:45:17.529083 18004 tablet_bootstrap.cc:492] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: No bootstrap required, opened a new log
I20260927 10:45:17.529112 18003 tablet_bootstrap.cc:492] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: No bootstrap required, opened a new log
I20260927 10:45:17.529273 18003 ts_tablet_manager.cc:1397] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:45:17.529285 18004 ts_tablet_manager.cc:1397] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 10:45:17.532967 18004 raft_consensus.cc:348] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.532958 18002 raft_consensus.cc:348] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.533222 18002 raft_consensus.cc:374] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:17.533224 18004 raft_consensus.cc:374] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:17.533291 18002 raft_consensus.cc:729] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfba3370ab7e43be9d07571f4d24e5cd, State: Initialized, Role: FOLLOWER
I20260927 10:45:17.533291 18004 raft_consensus.cc:729] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d618480d3ce74fd1b605fa2c691ff70c, State: Initialized, Role: FOLLOWER
I20260927 10:45:17.533591 18003 raft_consensus.cc:348] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.533813 18003 raft_consensus.cc:374] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:17.533773 18002 consensus_queue.cc:260] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.533869 18003 raft_consensus.cc:729] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbbf06c99e94029b60d7f9a26fda666, State: Initialized, Role: FOLLOWER
I20260927 10:45:17.533785 18004 consensus_queue.cc:260] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.534256 18003 consensus_queue.cc:260] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.534742 18002 ts_tablet_manager.cc:1428] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260927 10:45:17.534757 18004 ts_tablet_manager.cc:1428] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260927 10:45:17.535051 18003 ts_tablet_manager.cc:1428] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:45:17.535169 17970 heartbeater.cc:503] Master 127.10.96.254:32875 was elected leader, sending a full tablet report...
I20260927 10:45:17.536329 17697 heartbeater.cc:503] Master 127.10.96.254:32875 was elected leader, sending a full tablet report...
I20260927 10:45:17.538347 17834 heartbeater.cc:503] Master 127.10.96.254:32875 was elected leader, sending a full tablet report...
W20260927 10:45:17.580407 17699 tablet.cc:2369] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:45:17.717309 17972 tablet.cc:2369] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:45:17.773908 17836 tablet.cc:2369] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:45:17.787364 18008 raft_consensus.cc:482] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:17.787611 18008 raft_consensus.cc:504] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.788779 18008 leader_election.cc:288] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993), d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571)
I20260927 10:45:17.793762 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.789585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:37611 (local address 127.10.96.193:35571)
0927 10:45:17.789719 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.789724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.789741 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:45:17.789796 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.789799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.789849 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.789932 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:17.790296 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.790993 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.792454 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.793069 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.793198 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.793252 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.793539 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.793614 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:45:17.793748 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.789232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56953 (local address 127.10.96.194:44993)
0927 10:45:17.789395 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.789400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.789416 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:45:17.789597 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.789600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.789655 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.789726 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:17.790047 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.790789 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.792454 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.793068 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.793199 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.793253 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.793547 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.793614 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:45:17.793869 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.789455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35571 (local address 127.10.96.195:37611)
0927 10:45:17.789601 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.789612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.789715 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.790041 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.790046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.790067 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:17.790177 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.790181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.791101 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.791104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.792306 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.792316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.793202 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.793206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.793343 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.793405 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.793523 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.793682 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":117}
I20260927 10:45:17.793869 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.789084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:44993 (local address 127.10.96.195:56953)
0927 10:45:17.789305 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:17.789324 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:17.789438 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:17.789758 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:17.789761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:17.789777 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:17.789911 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.789917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.790918 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.790920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.792306 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:17.792316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.793202 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.793206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:17.793352 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.793410 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 10:45:17.793531 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 10:45:17.793682 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":126}
I20260927 10:45:17.794636 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d35d9ea3074119a07945f441b71e2a" candidate_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbbf06c99e94029b60d7f9a26fda666" is_pre_election: true
I20260927 10:45:17.794999 17779 raft_consensus.cc:2454] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfba3370ab7e43be9d07571f4d24e5cd in term 0.
I20260927 10:45:17.795131 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d35d9ea3074119a07945f441b71e2a" candidate_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d618480d3ce74fd1b605fa2c691ff70c" is_pre_election: true
I20260927 10:45:17.795425 17650 raft_consensus.cc:2454] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfba3370ab7e43be9d07571f4d24e5cd in term 0.
I20260927 10:45:17.795531 17859 leader_election.cc:302] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbbf06c99e94029b60d7f9a26fda666, dfba3370ab7e43be9d07571f4d24e5cd; no voters:
I20260927 10:45:17.795838 18008 raft_consensus.cc:2781] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:17.795939 18008 raft_consensus.cc:482] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:17.796021 18008 raft_consensus.cc:3037] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:17.797087 18008 raft_consensus.cc:504] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.797590 18008 leader_election.cc:288] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [CANDIDATE]: Term 1 election: Requested vote from peers 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993), d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571)
I20260927 10:45:17.797892 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d35d9ea3074119a07945f441b71e2a" candidate_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbbf06c99e94029b60d7f9a26fda666"
I20260927 10:45:17.797971 17650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d35d9ea3074119a07945f441b71e2a" candidate_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d618480d3ce74fd1b605fa2c691ff70c"
I20260927 10:45:17.798035 17779 raft_consensus.cc:3037] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:17.798077 17650 raft_consensus.cc:3037] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:17.799131 17650 raft_consensus.cc:2454] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfba3370ab7e43be9d07571f4d24e5cd in term 1.
I20260927 10:45:17.799144 17779 raft_consensus.cc:2454] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfba3370ab7e43be9d07571f4d24e5cd in term 1.
I20260927 10:45:17.799427 17859 leader_election.cc:302] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbbf06c99e94029b60d7f9a26fda666, dfba3370ab7e43be9d07571f4d24e5cd; no voters:
I20260927 10:45:17.799613 18008 raft_consensus.cc:2781] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:17.799841 18008 raft_consensus.cc:686] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 1 LEADER]: Becoming Leader. State: Replica: dfba3370ab7e43be9d07571f4d24e5cd, State: Running, Role: LEADER
I20260927 10:45:17.800199 18008 consensus_queue.cc:237] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } }
I20260927 10:45:17.803623 16734 catalog_manager.cc:5495] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd reported cstate change: term changed from 0 to 1, leader changed from <none> to dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195). New cstate: current_term: 1 leader_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:17.873567 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.870310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.10.96.195:38787)
0927 10:45:17.870434 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.870438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.870477 (+ 39us) server_negotiation.cc:407] Connection header received
0927 10:45:17.870548 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.870550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.870601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.870684 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:17.870893 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.871391 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.872176 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.872395 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.872474 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.872750 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:17.873230 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:17.873321 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.873392 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.873440 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:45:17.879612 17650 raft_consensus.cc:1261] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 1 FOLLOWER]: Refusing update from remote peer dfba3370ab7e43be9d07571f4d24e5cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:17.879616 17779 raft_consensus.cc:1261] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Refusing update from remote peer dfba3370ab7e43be9d07571f4d24e5cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:17.880360 18008 consensus_queue.cc:1035] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:17.880570 18011 consensus_queue.cc:1035] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:17.888403 18016 mvcc.cc:205] Tried to move back new op lower bound from 7333912239627587584 to 7333912239312367616. Current Snapshot: MvccSnapshot[applied={T|T < 7333912239627587584}]
I20260927 10:45:17.890823 18015 mvcc.cc:205] Tried to move back new op lower bound from 7333912239627587584 to 7333912239312367616. Current Snapshot: MvccSnapshot[applied={T|T < 7333912239627587584 or (T in {7333912239627587584})}]
I20260927 10:45:17.898406 16734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:17.898586 16734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:17.903738 16734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2231c3f1a56a41c0965b1e7bf3b3ad02: Sending DeleteTablet for 3 replicas of tablet b2d35d9ea3074119a07945f441b71e2a
I20260927 10:45:17.904701 17767 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d35d9ea3074119a07945f441b71e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:17 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:52840
I20260927 10:45:17.904821 17903 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d35d9ea3074119a07945f441b71e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:17 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:51818
I20260927 10:45:17.904861 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d35d9ea3074119a07945f441b71e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:17 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:38610
I20260927 10:45:17.905200 18024 tablet_replica.cc:330] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: stopping tablet replica
I20260927 10:45:17.905253 18025 tablet_replica.cc:330] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: stopping tablet replica
I20260927 10:45:17.905620 18024 raft_consensus.cc:2229] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:17.905622 18025 raft_consensus.cc:2229] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:17.905974 18025 raft_consensus.cc:2258] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:17.905985 18024 raft_consensus.cc:2258] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:17.907562 18024 ts_tablet_manager.cc:1905] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:17.907824 18025 ts_tablet_manager.cc:1905] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:17.908106 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 16702
I20260927 10:45:17.911440 18024 ts_tablet_manager.cc:1918] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:17.911612 18024 log.cc:1196] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/b2d35d9ea3074119a07945f441b71e2a
I20260927 10:45:17.912022 18024 ts_tablet_manager.cc:1939] T b2d35d9ea3074119a07945f441b71e2a P 1dbbf06c99e94029b60d7f9a26fda666: Deleting consensus metadata
I20260927 10:45:17.912007 18025 ts_tablet_manager.cc:1918] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:17.912103 18025 log.cc:1196] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/b2d35d9ea3074119a07945f441b71e2a
I20260927 10:45:17.912461 18025 ts_tablet_manager.cc:1939] T b2d35d9ea3074119a07945f441b71e2a P d618480d3ce74fd1b605fa2c691ff70c: Deleting consensus metadata
W20260927 10:45:17.915696 17584 connection.cc:535] server connection from 127.0.0.1:38610 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 10:45:17.916527 17723 connection.cc:535] server connection from 127.0.0.1:52840 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:45:17.917040 18026 tablet_replica.cc:330] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: stopping tablet replica
I20260927 10:45:17.917421 18026 raft_consensus.cc:2229] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:17.917971 18026 raft_consensus.cc:2258] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:17.920392 18026 ts_tablet_manager.cc:1905] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:17.923840 18026 ts_tablet_manager.cc:1918] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:17.923960 18026 log.cc:1196] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/b2d35d9ea3074119a07945f441b71e2a
I20260927 10:45:17.924343 18026 ts_tablet_manager.cc:1939] T b2d35d9ea3074119a07945f441b71e2a P dfba3370ab7e43be9d07571f4d24e5cd: Deleting consensus metadata
W20260927 10:45:17.925406 17858 connection.cc:441] server connection from 127.0.0.1:51818 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51818 (request call id 1) could send its response
I20260927 10:45:17.926277 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.906938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.10.96.253:42561)
0927 10:45:17.907169 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.907174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.916626 (+ 9452us) server_negotiation.cc:407] Connection header received
0927 10:45:17.916815 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.916819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.916875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.916955 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.918111 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.918916 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.920115 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.920441 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.920518 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.922508 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.922513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.922516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.922542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.924294 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.924844 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.924848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.924850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.924929 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.925203 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.925206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.925208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.925442 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.925585 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.925979 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.926092 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260927 10:45:17.934695 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.919662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.10.96.253:42561)
0927 10:45:17.919776 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.919781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.920994 (+ 1213us) server_negotiation.cc:407] Connection header received
0927 10:45:17.921575 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.921579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.921630 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.921685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.924438 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.925162 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.927971 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.928336 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.928413 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.930207 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.930212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.930216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.930254 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.932494 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.932988 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.932995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.932997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.933087 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.933689 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.933693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.933695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.933912 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.934029 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.934379 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.934508 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:45:17.935693 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.907044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.10.96.252:39443)
0927 10:45:17.907169 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.907174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.926309 (+ 19135us) server_negotiation.cc:407] Connection header received
0927 10:45:17.926803 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.926806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.926854 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.926903 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.927676 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.928429 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.929810 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.930153 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.930221 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.931759 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.931763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.931765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.931795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.933824 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.934323 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.934326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.934328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.934407 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.934694 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.934697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.934699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.934915 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.935013 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.935423 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.935550 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":4480}
I20260927 10:45:17.936055 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:17.920408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.10.96.252:39443)
0927 10:45:17.920531 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:17.920535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:17.921490 (+ 955us) server_negotiation.cc:407] Connection header received
0927 10:45:17.922765 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:17.922769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:17.922820 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:17.922887 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:17.926269 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.926962 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.927939 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:17.928285 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:17.928382 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:17.930721 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:17.930724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:17.930727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:17.930751 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:17.933718 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.934207 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.934210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.934212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.934299 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:17.934590 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:17.934593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:17.934595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:17.934796 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:17.934900 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:17.935441 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 10:45:17.935890 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":468,"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":12544}
I20260927 10:45:18.891398 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:18.890816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:48209)
0927 10:45:18.891183 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:18.891281 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":76,"threads_started":1}
W20260927 10:45:18.891901 17834 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:32875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:18.892184 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:18.891794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:58233)
0927 10:45:18.892023 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:18.892107 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
W20260927 10:45:18.892468 17970 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:32875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:18.893893 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:18.893568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:56773)
0927 10:45:18.893754 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:18.893822 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":54,"threads_started":1}
W20260927 10:45:18.894122 17697 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:32875 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:19.404052 18038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2231c3f1a56a41c0965b1e7bf3b3ad02)
I20260927 10:45:19.404240 18038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:19.404740 18039 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2231c3f1a56a41c0965b1e7bf3b3ad02)
I20260927 10:45:19.404917 18039 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:19.405323 18038 leader_election.cc:288] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875), f92005e8c3444f3e89f6633ce036c20d (127.10.96.252:39443)
I20260927 10:45:19.405927 18039 leader_election.cc:288] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875), 53234ec893754ca48c2936400df84c38 (127.10.96.253:42561)
I20260927 10:45:19.405963 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.405498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47414)
0927 10:45:19.405769 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.405853 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
W20260927 10:45:19.406143 17322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:19.406363 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.406003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47420)
0927 10:45:19.406177 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.406245 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
W20260927 10:45:19.406491 17504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:19.407058 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.406803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47434)
0927 10:45:19.406926 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.406980 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 10:45:19.407161 17322 leader_election.cc:334] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:19.407390 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.407079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47438)
0927 10:45:19.407264 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.407319 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 10:45:19.407513 17504 leader_election.cc:334] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:19.411080 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.406279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:42561 (local address 127.0.0.1:60136)
0927 10:45:19.406469 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.406481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:19.406579 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:19.407292 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:19.407295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:19.407309 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:19.407406 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:19.407412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.408453 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.408456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:19.409678 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:19.409688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.410568 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.410572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:19.410716 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:19.410741 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:45:19.410829 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:45:19.410952 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":71,"thread_start_us":57,"threads_started":1}
I20260927 10:45:19.411095 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.405550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:39443 (local address 127.0.0.1:60246)
0927 10:45:19.406692 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.406708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:19.406798 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:19.407168 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:19.407171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:19.407187 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:19.407301 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:19.407308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.408343 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.408346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:19.409677 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:19.409688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.410568 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.410572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:19.410715 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:19.410741 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:45:19.410823 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:45:19.410922 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":24,"thread_start_us":70,"threads_started":1}
I20260927 10:45:19.411106 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.406366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.10.96.253:42561)
0927 10:45:19.406830 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:19.406839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:19.406852 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:19.406898 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:19.406901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:19.406945 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:19.407030 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:19.407620 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.408360 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.409818 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.410453 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.410556 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:19.410674 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:19.410851 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:45:19.410958 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":401,"thread_start_us":74,"threads_started":1}
I20260927 10:45:19.411094 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.405898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.10.96.252:39443)
0927 10:45:19.406134 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:19.406139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:19.406881 (+ 742us) server_negotiation.cc:407] Connection header received
0927 10:45:19.406926 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:19.406929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:19.406975 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:19.407053 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:19.407402 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.408225 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.409827 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:19.410435 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:19.410556 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:19.410688 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:19.410851 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 10:45:19.410970 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
I20260927 10:45:19.411712 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f92005e8c3444f3e89f6633ce036c20d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53234ec893754ca48c2936400df84c38" is_pre_election: true
I20260927 10:45:19.411881 17344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f92005e8c3444f3e89f6633ce036c20d in term 1.
I20260927 10:45:19.412248 17504 leader_election.cc:302] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53234ec893754ca48c2936400df84c38, f92005e8c3444f3e89f6633ce036c20d; no voters: 2231c3f1a56a41c0965b1e7bf3b3ad02
I20260927 10:45:19.412262 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53234ec893754ca48c2936400df84c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f92005e8c3444f3e89f6633ce036c20d" is_pre_election: true
I20260927 10:45:19.412469 18039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:45:19.412549 17526 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53234ec893754ca48c2936400df84c38 in term 1.
I20260927 10:45:19.412599 18039 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2231c3f1a56a41c0965b1e7bf3b3ad02)
I20260927 10:45:19.412647 18039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:45:19.412864 17321 leader_election.cc:302] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53234ec893754ca48c2936400df84c38, f92005e8c3444f3e89f6633ce036c20d; no voters: 2231c3f1a56a41c0965b1e7bf3b3ad02
I20260927 10:45:19.413086 18038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:45:19.413142 18038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2231c3f1a56a41c0965b1e7bf3b3ad02)
I20260927 10:45:19.413190 18038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:45:19.413955 18039 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:19.414268 18039 leader_election.cc:288] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 2 election: Requested vote from peers 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875), 53234ec893754ca48c2936400df84c38 (127.10.96.253:42561)
I20260927 10:45:19.414270 18038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:19.414626 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.414405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47450)
0927 10:45:19.414493 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.414567 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 10:45:19.414736 18038 leader_election.cc:288] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [CANDIDATE]: Term 2 election: Requested vote from peers 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875), f92005e8c3444f3e89f6633ce036c20d (127.10.96.252:39443)
I20260927 10:45:19.414736 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f92005e8c3444f3e89f6633ce036c20d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53234ec893754ca48c2936400df84c38"
I20260927 10:45:19.415009 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.414766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47466)
0927 10:45:19.414890 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.414941 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 10:45:19.415145 17526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53234ec893754ca48c2936400df84c38" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f92005e8c3444f3e89f6633ce036c20d"
I20260927 10:45:19.415324 17526 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 53234ec893754ca48c2936400df84c38 in current term 2: Already voted for candidate f92005e8c3444f3e89f6633ce036c20d in this term.
I20260927 10:45:19.414948 17344 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f92005e8c3444f3e89f6633ce036c20d in current term 2: Already voted for candidate 53234ec893754ca48c2936400df84c38 in this term.
I20260927 10:45:19.415608 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.415363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47478)
0927 10:45:19.415505 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.415551 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 10:45:19.415608 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.415358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47468)
0927 10:45:19.415505 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.415551 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 10:45:19.415697 17322 leader_election.cc:334] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
W20260927 10:45:19.415701 17504 leader_election.cc:334] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:19.415759 17322 leader_election.cc:302] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53234ec893754ca48c2936400df84c38; no voters: 2231c3f1a56a41c0965b1e7bf3b3ad02, f92005e8c3444f3e89f6633ce036c20d
I20260927 10:45:19.415876 18038 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:45:19.415879 17504 leader_election.cc:302] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f92005e8c3444f3e89f6633ce036c20d; no voters: 2231c3f1a56a41c0965b1e7bf3b3ad02, 53234ec893754ca48c2936400df84c38
I20260927 10:45:19.416030 18039 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 10:45:19.892961 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.892433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:34661)
0927 10:45:19.892761 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.892840 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":94,"threads_started":1}
I20260927 10:45:19.893630 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.893095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:52181)
0927 10:45:19.893438 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.893534 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":64,"threads_started":1}
I20260927 10:45:19.894882 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:19.894567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:46561)
0927 10:45:19.894762 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:19.894813 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260927 10:45:20.894210 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:20.893684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:33785)
0927 10:45:20.894027 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:20.894100 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":97,"threads_started":1}
I20260927 10:45:20.895087 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:20.894670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:37943)
0927 10:45:20.894961 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:20.895012 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":70,"threads_started":1}
I20260927 10:45:20.895738 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:20.895427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:52083)
0927 10:45:20.895620 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:20.895669 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":53,"threads_started":1}
I20260927 10:45:21.091234 18054 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:21.091375 18054 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:21.091856 18054 leader_election.cc:288] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875), 53234ec893754ca48c2936400df84c38 (127.10.96.253:42561)
I20260927 10:45:21.092496 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.092087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47708)
0927 10:45:21.092339 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.092416 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260927 10:45:21.092646 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f92005e8c3444f3e89f6633ce036c20d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53234ec893754ca48c2936400df84c38" is_pre_election: true
I20260927 10:45:21.092815 17344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f92005e8c3444f3e89f6633ce036c20d in term 2.
I20260927 10:45:21.093369 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.093172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47720)
0927 10:45:21.093273 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.093319 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 10:45:21.093392 17504 leader_election.cc:302] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53234ec893754ca48c2936400df84c38, f92005e8c3444f3e89f6633ce036c20d; no voters:
W20260927 10:45:21.093573 17504 leader_election.cc:334] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:21.093573 18054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 10:45:21.093673 18054 raft_consensus.cc:482] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:21.093720 18054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:45:21.095059 18054 raft_consensus.cc:504] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:21.095417 18054 leader_election.cc:288] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 3 election: Requested vote from peers 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875), 53234ec893754ca48c2936400df84c38 (127.10.96.253:42561)
I20260927 10:45:21.095856 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.095592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47728)
0927 10:45:21.095718 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.095793 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 10:45:21.096006 17344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f92005e8c3444f3e89f6633ce036c20d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53234ec893754ca48c2936400df84c38"
I20260927 10:45:21.096130 17344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:45:21.097494 17344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f92005e8c3444f3e89f6633ce036c20d in term 3.
I20260927 10:45:21.097863 17504 leader_election.cc:302] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53234ec893754ca48c2936400df84c38, f92005e8c3444f3e89f6633ce036c20d; no voters:
I20260927 10:45:21.098021 18054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:45:21.098261 18054 raft_consensus.cc:686] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [term 3 LEADER]: Becoming Leader. State: Replica: f92005e8c3444f3e89f6633ce036c20d, State: Running, Role: LEADER
I20260927 10:45:21.098394 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.098187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47730)
0927 10:45:21.098284 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.098341 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 10:45:21.098560 17504 leader_election.cc:334] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
I20260927 10:45:21.098572 18054 consensus_queue.cc:237] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } }
I20260927 10:45:21.100008 18057 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f92005e8c3444f3e89f6633ce036c20d. Latest consensus state: current_term: 3 leader_uuid: "f92005e8c3444f3e89f6633ce036c20d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:21.100121 18057 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:21.100435 18059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:21.102623 18059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:21.102952 18059 catalog_manager.cc:1259] Loaded cluster ID: 77ca7f35a69446c39f57a0db1f742941
I20260927 10:45:21.102993 18059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:21.103499 18059 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:21.103828 18059 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Loaded TSK: 0
I20260927 10:45:21.104444 18059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:21.146528 17509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:21.149789 17509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:21.152645 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.152406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47750)
0927 10:45:21.152536 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.152583 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 10:45:21.152634 17344 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [term 3 FOLLOWER]: Refusing update from remote peer f92005e8c3444f3e89f6633ce036c20d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 10:45:21.153131 18054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 10:45:21.153432 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.153232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.0.0.1:47752)
0927 10:45:21.153332 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.153379 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:45:21.153611 17504 consensus_peers.cc:487] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d -> Peer 2231c3f1a56a41c0965b1e7bf3b3ad02 (127.10.96.254:32875): Couldn't send request to peer 2231c3f1a56a41c0965b1e7bf3b3ad02. Status: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 10:45:21.154592 18061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f92005e8c3444f3e89f6633ce036c20d. Latest consensus state: current_term: 3 leader_uuid: "f92005e8c3444f3e89f6633ce036c20d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:21.154733 18061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:21.155859 18057 sys_catalog.cc:455] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f92005e8c3444f3e89f6633ce036c20d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:21.155977 18057 sys_catalog.cc:458] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:21.156129 18061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f92005e8c3444f3e89f6633ce036c20d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2231c3f1a56a41c0965b1e7bf3b3ad02" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32875 } } peers { permanent_uuid: "53234ec893754ca48c2936400df84c38" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 42561 } attrs { promote: false } } peers { permanent_uuid: "f92005e8c3444f3e89f6633ce036c20d" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 39443 } attrs { promote: false } } }
I20260927 10:45:21.156268 18061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53234ec893754ca48c2936400df84c38 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:45:21.166836 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.162482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:44993 (local address 127.0.0.1:52850)
0927 10:45:21.162589 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.162606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:21.162699 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:21.163220 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:21.163224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:21.163248 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:21.163375 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.163385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.164585 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.164588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.165706 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.165714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.166433 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.166437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.166522 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.166544 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:45:21.166633 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:45:21.166704 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":34}
I20260927 10:45:21.166848 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.162588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.10.96.194:44993)
0927 10:45:21.162926 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.162932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.162946 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:21.162997 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.163003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.163066 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.163499 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:21.163620 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.164476 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.165822 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.166329 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.166432 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.166495 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.166661 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.166735 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":89,"threads_started":1}
I20260927 10:45:21.167366 17767 tablet_service.cc:1467] Processing CreateTablet for tablet d9f8268439ca4c998df7a496a4826d59 (DEFAULT_TABLE table=test-table [id=4d095cb48c044710a10e713d60c341f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:21.167678 17767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f8268439ca4c998df7a496a4826d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:21.168140 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.163186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:38787 (local address 127.0.0.1:51836)
0927 10:45:21.163641 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.163654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:21.163736 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:21.164247 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:21.164250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:21.164268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:21.164354 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.164359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.165423 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.165427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.166655 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.166665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.167646 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.167651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.167820 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.167844 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:45:21.167922 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:45:21.168002 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260927 10:45:21.168202 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.163391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.10.96.195:38787)
0927 10:45:21.163934 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.163939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.163955 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:45:21.164006 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.164009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.164055 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.164139 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:21.164482 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.165075 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.166812 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.167535 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.167654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.167721 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.167964 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.168048 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":83,"threads_started":1}
I20260927 10:45:21.169019 17903 tablet_service.cc:1467] Processing CreateTablet for tablet d9f8268439ca4c998df7a496a4826d59 (DEFAULT_TABLE table=test-table [id=4d095cb48c044710a10e713d60c341f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:21.169270 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.162734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35571 (local address 127.0.0.1:38620)
0927 10:45:21.162991 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.163003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:21.163103 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:21.164138 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:21.164140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:21.164155 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:21.164244 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.164252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.165215 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.165219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.167531 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.167540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.168592 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.168596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.168725 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.168744 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:45:21.168820 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:45:21.168887 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":81,"thread_start_us":102,"threads_started":1}
I20260927 10:45:21.169323 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f8268439ca4c998df7a496a4826d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:21.170251 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.163707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.10.96.193:35571)
0927 10:45:21.163933 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.163936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.163947 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:45:21.163987 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.163989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.164024 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.164116 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:21.164359 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.165071 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.167676 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.168392 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.169944 (+ 1552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.170008 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.170047 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.170119 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20260927 10:45:21.170444 18071 tablet_bootstrap.cc:492] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: Bootstrap starting.
I20260927 10:45:21.170759 17630 tablet_service.cc:1467] Processing CreateTablet for tablet d9f8268439ca4c998df7a496a4826d59 (DEFAULT_TABLE table=test-table [id=4d095cb48c044710a10e713d60c341f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:21.171010 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f8268439ca4c998df7a496a4826d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:21.171331 18071 tablet_bootstrap.cc:654] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:21.171978 18072 tablet_bootstrap.cc:492] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Bootstrap starting.
I20260927 10:45:21.172719 18071 tablet_bootstrap.cc:492] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: No bootstrap required, opened a new log
I20260927 10:45:21.172811 18071 ts_tablet_manager.cc:1397] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 10:45:21.172930 18072 tablet_bootstrap.cc:654] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:21.173257 18071 raft_consensus.cc:348] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.173372 18071 raft_consensus.cc:374] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:21.173410 18071 raft_consensus.cc:729] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbbf06c99e94029b60d7f9a26fda666, State: Initialized, Role: FOLLOWER
I20260927 10:45:21.173609 18071 consensus_queue.cc:260] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.173671 18074 tablet_bootstrap.cc:492] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c: Bootstrap starting.
I20260927 10:45:21.173972 18071 ts_tablet_manager.cc:1428] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:45:21.174192 18072 tablet_bootstrap.cc:492] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: No bootstrap required, opened a new log
I20260927 10:45:21.174280 18072 ts_tablet_manager.cc:1397] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:21.174681 18072 raft_consensus.cc:348] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.174817 18072 raft_consensus.cc:374] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:21.174854 18072 raft_consensus.cc:729] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfba3370ab7e43be9d07571f4d24e5cd, State: Initialized, Role: FOLLOWER
I20260927 10:45:21.174986 18074 tablet_bootstrap.cc:654] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:21.175009 18072 consensus_queue.cc:260] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.175396 18072 ts_tablet_manager.cc:1428] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:21.176231 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.175970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:58541)
0927 10:45:21.176095 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.176167 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 10:45:21.176275 18074 tablet_bootstrap.cc:492] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c: No bootstrap required, opened a new log
I20260927 10:45:21.176360 18074 ts_tablet_manager.cc:1397] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:45:21.176844 18074 raft_consensus.cc:348] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.176972 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.176673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:43427)
0927 10:45:21.176844 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.176902 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 10:45:21.176963 18074 raft_consensus.cc:374] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:21.177007 18074 raft_consensus.cc:729] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d618480d3ce74fd1b605fa2c691ff70c, State: Initialized, Role: FOLLOWER
I20260927 10:45:21.177174 18074 consensus_queue.cc:260] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.177603 18074 ts_tablet_manager.cc:1428] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 10:45:21.178364 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.178066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:49815)
0927 10:45:21.178233 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.178292 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 10:45:21.179353 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.177878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:36121)
0927 10:45:21.179241 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.179302 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1296}
I20260927 10:45:21.179365 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.179075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:57349)
0927 10:45:21.179226 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.179280 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 10:45:21.179682 17696 heartbeater.cc:503] Master 127.10.96.252:39443 was elected leader, sending a full tablet report...
I20260927 10:45:21.179777 17969 heartbeater.cc:503] Master 127.10.96.252:39443 was elected leader, sending a full tablet report...
I20260927 10:45:21.180352 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.179971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:59549)
0927 10:45:21.180233 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.180291 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 10:45:21.181285 18079 raft_consensus.cc:482] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:21.181416 18079 raft_consensus.cc:504] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.182617 18079 leader_election.cc:288] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993), dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787)
I20260927 10:45:21.187908 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.183025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:50091 (local address 127.10.96.195:38787)
0927 10:45:21.183140 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.183144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.183566 (+ 422us) server_negotiation.cc:407] Connection header received
0927 10:45:21.183604 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.183606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.183637 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.183741 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:21.183880 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.184663 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.186511 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.187121 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.187250 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.187296 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.187705 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.187789 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:45:21.187908 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.183235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55037 (local address 127.10.96.194:44993)
0927 10:45:21.183324 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.183327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.183338 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:45:21.183377 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.183379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.183413 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.183497 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:21.183741 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.184583 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.186511 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.187121 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.187249 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.187296 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.187708 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.187787 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 10:45:21.187950 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.182725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:44993 (local address 127.10.96.193:55037)
0927 10:45:21.182808 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.182822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:21.182906 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:21.183502 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:21.183504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:21.183515 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:21.183607 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.183611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.184699 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.184701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.186365 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.186375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.187265 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.187269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.187490 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.187569 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 10:45:21.187645 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:45:21.187797 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":256}
I20260927 10:45:21.187961 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.182943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:38787 (local address 127.10.96.193:50091)
0927 10:45:21.183199 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.183212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:21.183304 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:21.183739 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:21.183741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:21.183750 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:21.183812 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.183815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.184761 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.184763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.186365 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:21.186375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.187265 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.187269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:21.187492 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.187567 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 10:45:21.187645 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 10:45:21.187796 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":293,"thread_start_us":116,"threads_started":1}
I20260927 10:45:21.188416 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8268439ca4c998df7a496a4826d59" candidate_uuid: "d618480d3ce74fd1b605fa2c691ff70c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbbf06c99e94029b60d7f9a26fda666" is_pre_election: true
I20260927 10:45:21.188591 17779 raft_consensus.cc:2454] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d618480d3ce74fd1b605fa2c691ff70c in term 0.
I20260927 10:45:21.188997 17586 leader_election.cc:302] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dbbf06c99e94029b60d7f9a26fda666, d618480d3ce74fd1b605fa2c691ff70c; no voters:
I20260927 10:45:21.189093 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8268439ca4c998df7a496a4826d59" candidate_uuid: "d618480d3ce74fd1b605fa2c691ff70c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" is_pre_election: true
I20260927 10:45:21.189236 18079 raft_consensus.cc:2781] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:21.189301 18079 raft_consensus.cc:482] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:21.189356 18079 raft_consensus.cc:3037] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:21.189333 17923 raft_consensus.cc:2454] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d618480d3ce74fd1b605fa2c691ff70c in term 0.
I20260927 10:45:21.190440 18079 raft_consensus.cc:504] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.190742 18079 leader_election.cc:288] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [CANDIDATE]: Term 1 election: Requested vote from peers 1dbbf06c99e94029b60d7f9a26fda666 (127.10.96.194:44993), dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787)
I20260927 10:45:21.191072 17779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8268439ca4c998df7a496a4826d59" candidate_uuid: "d618480d3ce74fd1b605fa2c691ff70c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dbbf06c99e94029b60d7f9a26fda666"
I20260927 10:45:21.191149 17923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f8268439ca4c998df7a496a4826d59" candidate_uuid: "d618480d3ce74fd1b605fa2c691ff70c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfba3370ab7e43be9d07571f4d24e5cd"
I20260927 10:45:21.191226 17779 raft_consensus.cc:3037] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:21.191260 17923 raft_consensus.cc:3037] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:21.192343 17779 raft_consensus.cc:2454] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d618480d3ce74fd1b605fa2c691ff70c in term 1.
I20260927 10:45:21.192438 17923 raft_consensus.cc:2454] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d618480d3ce74fd1b605fa2c691ff70c in term 1.
I20260927 10:45:21.192725 17586 leader_election.cc:302] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d618480d3ce74fd1b605fa2c691ff70c, dfba3370ab7e43be9d07571f4d24e5cd; no voters:
I20260927 10:45:21.192919 18079 raft_consensus.cc:2781] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:21.193136 18079 raft_consensus.cc:686] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [term 1 LEADER]: Becoming Leader. State: Replica: d618480d3ce74fd1b605fa2c691ff70c, State: Running, Role: LEADER
I20260927 10:45:21.193413 18079 consensus_queue.cc:237] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } }
I20260927 10:45:21.193804 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.193601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:35645)
0927 10:45:21.193689 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.193747 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 10:45:21.195863 17513 catalog_manager.cc:5495] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c reported cstate change: term changed from 0 to 1, leader changed from <none> to d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193). New cstate: current_term: 1 leader_uuid: "d618480d3ce74fd1b605fa2c691ff70c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d618480d3ce74fd1b605fa2c691ff70c" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:21.208034 18070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.204774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.10.96.193:35571)
0927 10:45:21.204890 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:21.204893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:21.204920 (+ 27us) server_negotiation.cc:407] Connection header received
0927 10:45:21.205057 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:21.205059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:21.205106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:21.205182 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:21.205425 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.205984 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.206790 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:21.206996 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:21.207081 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:21.207274 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:21.207675 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:21.207761 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:21.207827 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 10:45:21.207870 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:21.210652 17779 raft_consensus.cc:1261] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Refusing update from remote peer d618480d3ce74fd1b605fa2c691ff70c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:21.210970 17923 raft_consensus.cc:1261] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Refusing update from remote peer d618480d3ce74fd1b605fa2c691ff70c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:21.211210 18079 consensus_queue.cc:1035] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dbbf06c99e94029b60d7f9a26fda666" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:21.211526 18081 consensus_queue.cc:1035] T d9f8268439ca4c998df7a496a4826d59 P d618480d3ce74fd1b605fa2c691ff70c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfba3370ab7e43be9d07571f4d24e5cd" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:21.212423 17833 heartbeater.cc:503] Master 127.10.96.252:39443 was elected leader, sending a full tablet report...
I20260927 10:45:21.214203 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.213898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:34363)
0927 10:45:21.214067 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.214124 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 10:45:21.214421 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.213818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:50591)
0927 10:45:21.214276 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.214333 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
I20260927 10:45:21.217263 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.216981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.195:47261)
0927 10:45:21.217128 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.217181 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 10:45:21.217854 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.217352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:51811)
0927 10:45:21.217737 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.217791 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
I20260927 10:45:21.218854 18086 mvcc.cc:205] Tried to move back new op lower bound from 7333912253273493504 to 7333912253211496448. Current Snapshot: MvccSnapshot[applied={T|T < 7333912253273493504 or (T in {7333912253273493504})}]
I20260927 10:45:21.219751 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.219555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.193:53173)
0927 10:45:21.219640 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.219694 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 10:45:21.220743 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.220468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32875 (local address 127.10.96.194:58371)
0927 10:45:21.220617 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.220671 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:32875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 10:45:21.225250 17511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:21.225376 17511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:21.228870 17511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f92005e8c3444f3e89f6633ce036c20d: Sending DeleteTablet for 3 replicas of tablet d9f8268439ca4c998df7a496a4826d59
I20260927 10:45:21.229492 17767 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f8268439ca4c998df7a496a4826d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:21 UTC) from {username='oryx', principal='oryx/127.10.96.252@KRBTEST.COM'} at 127.0.0.1:52850
I20260927 10:45:21.229790 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f8268439ca4c998df7a496a4826d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:21 UTC) from {username='oryx', principal='oryx/127.10.96.252@KRBTEST.COM'} at 127.0.0.1:38620
I20260927 10:45:21.229982 17903 tablet_service.cc:1514] Processing DeleteTablet for tablet d9f8268439ca4c998df7a496a4826d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:21 UTC) from {username='oryx', principal='oryx/127.10.96.252@KRBTEST.COM'} at 127.0.0.1:51836
I20260927 10:45:21.230096 18093 tablet_replica.cc:330] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: stopping tablet replica
I20260927 10:45:21.230211 18093 raft_consensus.cc:2229] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:21.230556 18095 tablet_replica.cc:330] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: stopping tablet replica
I20260927 10:45:21.230687 18095 raft_consensus.cc:2229] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:21.230814 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 17479
I20260927 10:45:21.230847 18095 raft_consensus.cc:2258] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:21.233057 18093 raft_consensus.cc:2258] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:21.233794 18095 ts_tablet_manager.cc:1905] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:21.235857 18095 ts_tablet_manager.cc:1918] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:21.235939 18095 log.cc:1196] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/d9f8268439ca4c998df7a496a4826d59
I20260927 10:45:21.235986 18093 ts_tablet_manager.cc:1905] T d9f8268439ca4c998df7a496a4826d59 P 1dbbf06c99e94029b60d7f9a26fda666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:21.236269 18095 ts_tablet_manager.cc:1939] T d9f8268439ca4c998df7a496a4826d59 P dfba3370ab7e43be9d07571f4d24e5cd: Deleting consensus metadata
I20260927 10:45:21.236990 17504 catalog_manager.cc:4841] TS dfba3370ab7e43be9d07571f4d24e5cd (127.10.96.195:38787): tablet d9f8268439ca4c998df7a496a4826d59 (table test-table [id=4d095cb48c044710a10e713d60c341f3]) successfully deleted
I20260927 10:45:21.237670 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 17704
I20260927 10:45:21.238000 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.237713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35571 (local address 127.0.0.1:38640)
0927 10:45:21.237860 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.237915 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:35571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 10:45:21.238174 17504 catalog_manager.cc:4584] TS d618480d3ce74fd1b605fa2c691ff70c (127.10.96.193:35571): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9f8268439ca4c998df7a496a4826d59: Network error: Client connection negotiation failed: client connection to 127.10.96.193:35571: connect: Connection refused (error 111)
I20260927 10:45:21.243631 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 17840
I20260927 10:45:21.243822 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:21.243565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:44993 (local address 127.0.0.1:52860)
0927 10:45:21.243682 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:21.243737 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:44993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:45:21.250881 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 17198
I20260927 10:45:21.256325 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 17301
2026-09-27T10:45:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7019 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:21 dist-test-slave-wmg0 krb5kdc[18098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:21 dist-test-slave-wmg0 krb5kdc[18098](info): set up 2 sockets
Sep 27 10:45:21 dist-test-slave-wmg0 krb5kdc[18098](info): commencing operation
krb5kdc: starting...
W20260927 10:45:22.282167 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:22 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab.
Entry for principal oryx/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab.
I20260927 10:45:22.399955 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:44143
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab
--principal=oryx/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:44143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:22.506206 18111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:22.506467 18111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:22.506508 18111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:22.510425 18111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:22.510471 18111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 10:45:22.510488 18111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:22.510507 18111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:22.510524 18111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:22.510537 18111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:22.515555 18111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:44143
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.254.keytab
--principal=oryx/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:44143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:22.516467 18111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:22.517405 18111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:22.525424 18117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.525460 18119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:22.525671 18111 server_base.cc:1034] running on GCE node
W20260927 10:45:22.525424 18116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:22.526517 18111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:22.527643 18111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:22.528844 18111 hybrid_clock.cc:648] HybridClock initialized: now 1790505922528811 us; error 44 us; skew 500 ppm
Sep 27 10:45:22 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:22.536646 18111 init.cc:377] Logged in from keytab as oryx/127.10.96.254@KRBTEST.COM (short username oryx)
I20260927 10:45:22.537695 18111 webserver.cc:460] Webserver started at http://127.10.96.254:46837/ using document root <none> and password file <none>
I20260927 10:45:22.538254 18111 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:22.538342 18111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:22.538573 18111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:22.540575 18111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "778717d896d74aa3b476e2ab08f6626d"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-wmg0"
server_key: "7baae4851f62ec46cb34a001779606fc"
server_key_iv: "fd9912a83047dbf1eda253230f6ac7a5"
server_key_version: "encryptionkey@0"
I20260927 10:45:22.541112 18111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "778717d896d74aa3b476e2ab08f6626d"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-wmg0"
server_key: "7baae4851f62ec46cb34a001779606fc"
server_key_iv: "fd9912a83047dbf1eda253230f6ac7a5"
server_key_version: "encryptionkey@0"
I20260927 10:45:22.544834 18111 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:45:22.547389 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:22.548588 18111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:22.548707 18111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "778717d896d74aa3b476e2ab08f6626d"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-wmg0"
server_key: "7baae4851f62ec46cb34a001779606fc"
server_key_iv: "fd9912a83047dbf1eda253230f6ac7a5"
server_key_version: "encryptionkey@0"
I20260927 10:45:22.548806 18111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:22.559669 18111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:22.562793 18111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:22.562987 18111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:22.570703 18111 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:44143
I20260927 10:45:22.570724 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:44143 every 8 connection(s)
I20260927 10:45:22.571606 18111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:22.574429 18189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:22.575488 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 18111
I20260927 10:45:22.575609 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:22.575924 10627 external_mini_cluster.cc:1428] Setting key 5180ceaf3548c66ce11e8a2b5dbc2cd6
I20260927 10:45:22.580823 18189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Bootstrap starting.
I20260927 10:45:22.583477 18189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:22.584293 18189 log.cc:824] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:45:22 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:22.586148 18189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: No bootstrap required, opened a new log
I20260927 10:45:22.588871 18189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778717d896d74aa3b476e2ab08f6626d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 44143 } }
I20260927 10:45:22.589151 18189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:22.589234 18189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778717d896d74aa3b476e2ab08f6626d, State: Initialized, Role: FOLLOWER
I20260927 10:45:22.589699 18189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778717d896d74aa3b476e2ab08f6626d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 44143 } }
I20260927 10:45:22.589818 18189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:22.589901 18189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:22.589999 18189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:22.590202 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.577184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.10.96.254:44143)
0927 10:45:22.577579 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:22.577599 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:22.577632 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:45:22.578248 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:22.578263 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:22.578563 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:22.578835 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:22.579669 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.580622 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.581727 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.582170 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.582287 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.585004 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:22.585008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:22.585021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:22.585046 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:22.587320 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.587762 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.587767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.587772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.587873 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.588144 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.588149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.588151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.588519 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:22.588722 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:22.588977 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 10:45:22.589264 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
I20260927 10:45:22.590906 18189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778717d896d74aa3b476e2ab08f6626d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 44143 } }
I20260927 10:45:22.591287 18189 leader_election.cc:302] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 778717d896d74aa3b476e2ab08f6626d; no voters:
I20260927 10:45:22.591580 18189 leader_election.cc:288] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:22.591693 18194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:22.591908 18194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [term 1 LEADER]: Becoming Leader. State: Replica: 778717d896d74aa3b476e2ab08f6626d, State: Running, Role: LEADER
I20260927 10:45:22.592234 18194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778717d896d74aa3b476e2ab08f6626d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 44143 } }
I20260927 10:45:22.592825 18189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:22.593297 18194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 778717d896d74aa3b476e2ab08f6626d. Latest consensus state: current_term: 1 leader_uuid: "778717d896d74aa3b476e2ab08f6626d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778717d896d74aa3b476e2ab08f6626d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 44143 } } }
I20260927 10:45:22.593526 18194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:22.593756 18196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "778717d896d74aa3b476e2ab08f6626d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "778717d896d74aa3b476e2ab08f6626d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 44143 } } }
I20260927 10:45:22.593909 18196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:22.593981 18200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:22.596899 18200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:22.602195 18200 catalog_manager.cc:1347] Generated new cluster ID: 009e9ec3a2f44c8986fa2e2e37df777a
I20260927 10:45:22.602252 18200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:22.618245 18200 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:22.619203 18200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:22.624269 18200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Generated new TSK 0
I20260927 10:45:22.624933 18200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab.
Entry for principal oryx/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab.
I20260927 10:45:22.658680 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:44143
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:22.771636 18215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:22.771881 18215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:22.771925 18215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:22.776062 18215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:22.776118 18215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:22.776155 18215 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:45:22.776227 18215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:22.781399 18215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:22.782469 18215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:22.783488 18215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:22.792735 18220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.792735 18221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:22.792758 18223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:22.793056 18215 server_base.cc:1034] running on GCE node
I20260927 10:45:22.793576 18215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:22.794260 18215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:22.795449 18215 hybrid_clock.cc:648] HybridClock initialized: now 1790505922795429 us; error 34 us; skew 500 ppm
Sep 27 10:45:22 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:22.798210 18215 init.cc:377] Logged in from keytab as oryx/127.10.96.193@KRBTEST.COM (short username oryx)
I20260927 10:45:22.799121 18215 webserver.cc:460] Webserver started at http://127.10.96.193:33597/ using document root <none> and password file <none>
I20260927 10:45:22.799703 18215 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:22.799794 18215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:22.800019 18215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:22.802112 18215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "137ab6c704184cbebdfadd3bbba9ad81"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-wmg0"
server_key: "138296385173fefa09e80273a62faa34"
server_key_iv: "5b4159b3fedca49dda7f57b5d96b0e21"
server_key_version: "encryptionkey@0"
I20260927 10:45:22.802677 18215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "137ab6c704184cbebdfadd3bbba9ad81"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-wmg0"
server_key: "138296385173fefa09e80273a62faa34"
server_key_iv: "5b4159b3fedca49dda7f57b5d96b0e21"
server_key_version: "encryptionkey@0"
I20260927 10:45:22.806414 18215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:45:22.808720 18230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:22.809926 18215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:22.810050 18215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "137ab6c704184cbebdfadd3bbba9ad81"
format_stamp: "Formatted at 2026-09-27 10:45:22 on dist-test-slave-wmg0"
server_key: "138296385173fefa09e80273a62faa34"
server_key_iv: "5b4159b3fedca49dda7f57b5d96b0e21"
server_key_version: "encryptionkey@0"
I20260927 10:45:22.810164 18215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:22.821080 18215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:22.823858 18215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:22.824046 18215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:22.825593 18215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:22.825659 18215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:22.825757 18215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:22.825794 18215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:45:22 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:22.836460 18215 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:34087
I20260927 10:45:22.836522 18345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:34087 every 8 connection(s)
I20260927 10:45:22.837389 18215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:22.839042 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.826226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55675 (local address 127.10.96.254:44143)
0927 10:45:22.826348 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:22.826352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:22.827174 (+ 822us) server_negotiation.cc:407] Connection header received
0927 10:45:22.828484 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:22.828488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:22.828554 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:22.828616 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:22.830494 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.831101 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.832261 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.832510 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.832625 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.835342 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:22.835347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:22.835350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:22.835381 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:22.836962 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.837479 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.837485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.837490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.837579 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.837954 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.837957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.837958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.838171 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:22.838291 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:22.838754 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 10:45:22.838878 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:45:22.839785 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.826561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.10.96.193:55675)
0927 10:45:22.827014 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:22.827071 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:22.828222 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:22.829067 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:22.829077 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:22.829621 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:22.830316 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:22.830337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.831269 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.831273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:22.832126 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:22.832134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.832671 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.832675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:22.832874 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.833536 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:22.833574 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:22.835137 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:22.837075 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:22.837082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:22.837095 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:22.837358 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:22.837671 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:22.837676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:22.837678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:22.837839 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:22.838287 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:22.838293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:22.838516 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 10:45:22.838693 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:45:22.838985 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260927 10:45:22.841022 18346 heartbeater.cc:347] Connected to a master server at 127.10.96.254:44143
I20260927 10:45:22.841181 18346 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:22.841730 18346 heartbeater.cc:511] Master 127.10.96.254:44143 requested a full tablet report, sending...
I20260927 10:45:22.843120 18142 ts_manager.cc:194] Registered new tserver with Master: 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087)
I20260927 10:45:22.844594 18142 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:55675
I20260927 10:45:22.845183 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 18215
I20260927 10:45:22.845335 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:22.845660 10627 external_mini_cluster.cc:1428] Setting key 39a8bc127b59d4d023c228598c05801e
I20260927 10:45:22.851082 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.843327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:43735 (local address 127.10.96.254:44143)
0927 10:45:22.843428 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:22.843432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:22.843624 (+ 192us) server_negotiation.cc:407] Connection header received
0927 10:45:22.843711 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:22.843714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:22.843754 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:22.843830 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:22.844195 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.844920 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.846177 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.846429 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.846540 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.847781 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:22.847784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:22.847786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:22.847807 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:22.849233 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.849774 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.849777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.849779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.849867 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:22.850171 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:22.850174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:22.850175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:22.850441 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:22.850547 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:22.850879 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 10:45:22.850972 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260927 10:45:22.851263 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:22.843265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.10.96.193:43735)
0927 10:45:22.843541 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:22.843553 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:22.843634 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:22.843842 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:22.843844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:22.843997 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:22.844090 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:22.844095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.845061 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.845067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:22.846056 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:22.846065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:22.846543 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:22.846546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:22.846635 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:22.847179 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:22.847194 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:22.847616 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:22.849347 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:22.849353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:22.849357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:22.849667 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:22.849969 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:22.849973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:22.849976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:22.850076 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:22.850552 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:22.850558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:22.850670 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 10:45:22.850794 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 10:45:22.850960 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":35,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab.
Entry for principal oryx/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab.
I20260927 10:45:22.873519 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:44143
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:22.980981 18355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:22.981225 18355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:22.981264 18355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:22.985174 18355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:22.985229 18355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:22.985265 18355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:45:22.985342 18355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:22.990636 18355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:22.991605 18355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:22.992578 18355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:23.002034 18361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:23.002120 18355 server_base.cc:1034] running on GCE node
W20260927 10:45:23.002034 18360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:23.002058 18363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:23.002753 18355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:23.003435 18355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:23.004647 18355 hybrid_clock.cc:648] HybridClock initialized: now 1790505923004611 us; error 46 us; skew 500 ppm
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:23.007753 18355 init.cc:377] Logged in from keytab as oryx/127.10.96.194@KRBTEST.COM (short username oryx)
I20260927 10:45:23.008760 18355 webserver.cc:460] Webserver started at http://127.10.96.194:33871/ using document root <none> and password file <none>
I20260927 10:45:23.009362 18355 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:23.009460 18355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:23.009752 18355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:23.011925 18355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "124b52055c774efba6b7f19e68cd133f"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-wmg0"
server_key: "f752106e96f8093024c4669453f2343c"
server_key_iv: "659c9984780bdc61364b5140fcc77386"
server_key_version: "encryptionkey@0"
I20260927 10:45:23.012558 18355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "124b52055c774efba6b7f19e68cd133f"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-wmg0"
server_key: "f752106e96f8093024c4669453f2343c"
server_key_iv: "659c9984780bdc61364b5140fcc77386"
server_key_version: "encryptionkey@0"
I20260927 10:45:23.016619 18355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 10:45:23.019248 18370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.020498 18355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:23.020634 18355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "124b52055c774efba6b7f19e68cd133f"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-wmg0"
server_key: "f752106e96f8093024c4669453f2343c"
server_key_iv: "659c9984780bdc61364b5140fcc77386"
server_key_version: "encryptionkey@0"
I20260927 10:45:23.020741 18355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:23.037231 18355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:23.040700 18355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:23.040901 18355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:23.042518 18355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:23.042590 18355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.042682 18355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:23.042726 18355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:23.053723 18355 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:34029
I20260927 10:45:23.054051 18485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:34029 every 8 connection(s)
I20260927 10:45:23.054864 18355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:23.056386 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.043162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:58193 (local address 127.10.96.254:44143)
0927 10:45:23.043296 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.043303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.044191 (+ 888us) server_negotiation.cc:407] Connection header received
0927 10:45:23.045153 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.045157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.045219 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.045324 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.046509 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.047390 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.048823 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.049100 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.049217 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.052574 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.052577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.052580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.052615 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.054394 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.054860 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.054864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.054866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.054962 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.055291 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.055296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.055297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.055527 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.055622 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.056130 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.056240 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:45:23.057081 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.043583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.10.96.194:58193)
0927 10:45:23.044023 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.044073 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.044913 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.045537 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.045545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.045918 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.046369 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.046382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.047527 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.047532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.048701 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.048710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.049271 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.049276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.049531 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.050251 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.050275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:23.052401 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:23.054504 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.054508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.054519 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.054756 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.055046 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.055049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.055051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.055192 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.055672 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:23.055678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:23.055907 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.056064 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.056294 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":122,"threads_started":1}
I20260927 10:45:23.058291 18486 heartbeater.cc:347] Connected to a master server at 127.10.96.254:44143
I20260927 10:45:23.058449 18486 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:23.058941 18486 heartbeater.cc:511] Master 127.10.96.254:44143 requested a full tablet report, sending...
I20260927 10:45:23.059526 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 18355
I20260927 10:45:23.059645 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:23.059928 10627 external_mini_cluster.cc:1428] Setting key dd783a44bcd2231a0eee4cbe79d81e16
I20260927 10:45:23.059921 18143 ts_manager.cc:194] Registered new tserver with Master: 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029)
I20260927 10:45:23.060623 18143 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:58193
WARNING: no policy specified for oryx/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:45:23.069808 18492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.060421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.10.96.194:45389)
0927 10:45:23.061095 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.061110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.062730 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.063008 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.063010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.063166 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.063254 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.063258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.064278 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.064280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.064831 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.064838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.065271 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.065275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.065390 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.065986 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.065997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:23.066305 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:23.068207 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.068209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.068211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.068397 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.068679 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.068681 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.068682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.068748 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.069262 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:23.069265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:23.069340 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.069425 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.069626 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":84,"threads_started":1}
I20260927 10:45:23.069854 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.060501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:45389 (local address 127.10.96.254:44143)
0927 10:45:23.060602 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.060607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.061168 (+ 561us) server_negotiation.cc:407] Connection header received
0927 10:45:23.062840 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.062846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.062899 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.062989 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.063355 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.064160 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.064945 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.065164 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.065270 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.066444 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.066451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.066457 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.066489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.068095 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.068489 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.068492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.068494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.068585 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.068853 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.068857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.068860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.069161 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.069266 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.069537 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.069664 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab.
Entry for principal oryx/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab.
I20260927 10:45:23.088061 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:44143
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:23.200548 18495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:23.200766 18495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:23.200836 18495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:23.204706 18495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:23.204756 18495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:23.204797 18495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 10:45:23.204829 18495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:23.209834 18495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.96.254:44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:23.210718 18495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:23.211645 18495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:23.220947 18503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:23.221009 18495 server_base.cc:1034] running on GCE node
W20260927 10:45:23.220952 18500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:23.220947 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:23.221625 18495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:23.222239 18495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:23.223392 18495 hybrid_clock.cc:648] HybridClock initialized: now 1790505923223378 us; error 25 us; skew 500 ppm
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:23.226100 18495 init.cc:377] Logged in from keytab as oryx/127.10.96.195@KRBTEST.COM (short username oryx)
I20260927 10:45:23.226944 18495 webserver.cc:460] Webserver started at http://127.10.96.195:46363/ using document root <none> and password file <none>
I20260927 10:45:23.227481 18495 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:23.227566 18495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:23.227794 18495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:23.229840 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-wmg0"
server_key: "ba5eb3a3b1f3250273bbdfa195c039ff"
server_key_iv: "65a2494c451531dc69812e799892b841"
server_key_version: "encryptionkey@0"
I20260927 10:45:23.230373 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-wmg0"
server_key: "ba5eb3a3b1f3250273bbdfa195c039ff"
server_key_iv: "65a2494c451531dc69812e799892b841"
server_key_version: "encryptionkey@0"
I20260927 10:45:23.233933 18495 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:45:23.236126 18510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.237284 18495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:23.237561 18495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
format_stamp: "Formatted at 2026-09-27 10:45:23 on dist-test-slave-wmg0"
server_key: "ba5eb3a3b1f3250273bbdfa195c039ff"
server_key_iv: "65a2494c451531dc69812e799892b841"
server_key_version: "encryptionkey@0"
I20260927 10:45:23.237668 18495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:23.249044 18495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:23.251823 18495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:23.252009 18495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:23.253423 18495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:23.253542 18495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:23.253612 18495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:23.253657 18495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:23.264513 18495 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:38951
I20260927 10:45:23.265707 18495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:23.266924 18625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:38951 every 8 connection(s)
I20260927 10:45:23.268461 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.254371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:36947 (local address 127.10.96.254:44143)
0927 10:45:23.254534 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.254541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.255470 (+ 929us) server_negotiation.cc:407] Connection header received
0927 10:45:23.256442 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.256446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.256506 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.256561 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.257934 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.258717 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.260664 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.261045 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.261159 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.264094 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.264098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.264103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.264131 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.266087 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.266671 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.266676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.266678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.266761 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.267202 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.267206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.267209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.267445 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.267538 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.268176 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.268303 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:45:23.269405 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.254731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.10.96.195:36947)
0927 10:45:23.255279 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.255339 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.256167 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.256898 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.256904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.257222 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.257794 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.257809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.258858 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.258865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.260515 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.260532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.261206 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.261210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.261408 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.262060 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.262077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:23.263912 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:23.266220 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.266226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.266239 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.266557 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.266869 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.266872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.266874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.267077 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.267592 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:23.267600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:23.267891 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.268091 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.268404 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":225,"threads_started":1}
I20260927 10:45:23.273181 18626 heartbeater.cc:347] Connected to a master server at 127.10.96.254:44143
I20260927 10:45:23.273345 18626 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:23.273880 18626 heartbeater.cc:511] Master 127.10.96.254:44143 requested a full tablet report, sending...
I20260927 10:45:23.274236 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 18495
I20260927 10:45:23.274363 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:23.274672 10627 external_mini_cluster.cc:1428] Setting key 907499899bd90f285991f58bbfea13d5
I20260927 10:45:23.274911 18143 ts_manager.cc:194] Registered new tserver with Master: 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:23.275643 18143 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:36947
I20260927 10:45:23.276613 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:45:23.282261 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.273197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:48007 (local address 127.10.96.254:44143)
0927 10:45:23.273296 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.273300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.273629 (+ 329us) server_negotiation.cc:407] Connection header received
0927 10:45:23.273732 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.273734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.273771 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.273833 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.274344 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.274954 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.276132 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.276363 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.276472 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.278794 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.278797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.278799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.278819 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.280508 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.281031 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.281037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.281041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.281138 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.281404 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.281408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.281411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.281668 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.281772 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.282052 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.282152 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 10:45:23.282339 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.273152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.10.96.195:48007)
0927 10:45:23.273537 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.273552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.273649 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.273872 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.273874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.274100 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.274213 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.274220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.275106 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.275109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.276023 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.276031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.277559 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.277563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.277652 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.278193 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.278204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:23.278662 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:23.280626 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.280629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.280630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.280911 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.281236 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:23.281238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:23.281239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:23.281312 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:23.281759 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:23.281761 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:23.281852 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.281967 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.282126 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":191,"threads_started":1}
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:45:23.301687 18192 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:23.295423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.10.96.254:44143)
0927 10:45:23.295584 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.295587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.295598 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:45:23.295635 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.295638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.295674 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.295770 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.296460 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.296994 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.297980 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.298198 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.298309 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.301569 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60502: BlockingRecv error: recv got EOF from 127.0.0.1:60502 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:23.311708 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.303439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.10.96.254:44143)
0927 10:45:23.303587 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.303591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.303709 (+ 118us) server_negotiation.cc:407] Connection header received
0927 10:45:23.303888 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.303895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.303935 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.304015 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.304634 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.305142 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.306072 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.306295 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.306402 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.308804 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:23.308808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:23.308811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:23.308840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:23.310140 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.310555 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.310560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.310562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.310637 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:23.310905 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:23.310909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:23.310911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:23.311079 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:23.311214 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.311435 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.311545 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:45:23.314330 18142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:23.317350 18142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:23.330953 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.325376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:38951 (local address 127.0.0.1:50628)
0927 10:45:23.325752 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.325802 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.325923 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.327082 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.327085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.327101 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.327184 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.327190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.328466 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.328471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.329647 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.329655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.330422 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.330426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.330536 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.330576 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.330684 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.330789 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":89,"thread_start_us":114,"threads_started":1}
I20260927 10:45:23.331091 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.325252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34087 (local address 127.0.0.1:51728)
0927 10:45:23.325743 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.325798 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.326103 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.326653 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.326658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.326681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.326815 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.326822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.328208 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.328211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.329682 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.329693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.330704 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.330707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.330796 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.330814 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.330873 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.330924 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":73,"thread_start_us":84,"threads_started":1}
I20260927 10:45:23.331437 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.325416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.10.96.193:34087)
0927 10:45:23.326007 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.326021 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.326046 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:45:23.326249 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.326254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.326482 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.326653 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.326945 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.328099 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.329826 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.330609 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.330709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.330861 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.330909 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.330991 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":115,"threads_started":1}
I20260927 10:45:23.331678 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.326057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.10.96.195:38951)
0927 10:45:23.326403 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.326421 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.326444 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:45:23.326597 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.326605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.326789 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.326935 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.327319 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.328360 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.329762 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.330305 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.330424 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.330588 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.331412 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.331538 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20260927 10:45:23.331933 18652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.326904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34029 (local address 127.0.0.1:54806)
0927 10:45:23.327184 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.327197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.327280 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.327753 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.327756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.327772 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.327882 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.327889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.329185 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.329188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.330353 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.330362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.331552 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.331557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.331653 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.331674 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.331741 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.331813 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":82,"threads_started":1}
I20260927 10:45:23.332402 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.327043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.10.96.194:34029)
0927 10:45:23.327429 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.327437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.327453 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:45:23.327508 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.327512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.327631 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.327777 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.327999 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.329061 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.330482 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.331236 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.331348 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.331487 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.332171 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.332266 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":95,"threads_started":1}
I20260927 10:45:23.334272 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 08c004f3fc7044019308f8cac44b7a0a (DEFAULT_TABLE table=test-table [id=67ceb66ddd9a45889e115478b3c4715b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.334472 18560 tablet_service.cc:1467] Processing CreateTablet for tablet 08c004f3fc7044019308f8cac44b7a0a (DEFAULT_TABLE table=test-table [id=67ceb66ddd9a45889e115478b3c4715b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.335250 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c004f3fc7044019308f8cac44b7a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.335079 18420 tablet_service.cc:1467] Processing CreateTablet for tablet 08c004f3fc7044019308f8cac44b7a0a (DEFAULT_TABLE table=test-table [id=67ceb66ddd9a45889e115478b3c4715b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.335479 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c004f3fc7044019308f8cac44b7a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.335942 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c004f3fc7044019308f8cac44b7a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.343133 18656 tablet_bootstrap.cc:492] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Bootstrap starting.
I20260927 10:45:23.343837 18654 tablet_bootstrap.cc:492] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Bootstrap starting.
I20260927 10:45:23.345038 18656 tablet_bootstrap.cc:654] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.345547 18654 tablet_bootstrap.cc:654] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.345803 18655 tablet_bootstrap.cc:492] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Bootstrap starting.
I20260927 10:45:23.346076 18656 log.cc:824] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:23.346335 18654 log.cc:824] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:23.347921 18655 tablet_bootstrap.cc:654] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.348050 18654 tablet_bootstrap.cc:492] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: No bootstrap required, opened a new log
I20260927 10:45:23.348127 18656 tablet_bootstrap.cc:492] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: No bootstrap required, opened a new log
I20260927 10:45:23.348237 18654 ts_tablet_manager.cc:1397] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 10:45:23.348309 18656 ts_tablet_manager.cc:1397] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:45:23.348688 18655 log.cc:824] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:23.350593 18655 tablet_bootstrap.cc:492] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: No bootstrap required, opened a new log
I20260927 10:45:23.350769 18655 ts_tablet_manager.cc:1397] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:45:23.351567 18654 raft_consensus.cc:348] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.351861 18654 raft_consensus.cc:374] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.351933 18654 raft_consensus.cc:729] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137ab6c704184cbebdfadd3bbba9ad81, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.352027 18656 raft_consensus.cc:348] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.352259 18656 raft_consensus.cc:374] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.352352 18656 raft_consensus.cc:729] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.352624 18654 consensus_queue.cc:260] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.352810 18656 consensus_queue.cc:260] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.353842 18654 ts_tablet_manager.cc:1428] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 10:45:23.353844 18656 ts_tablet_manager.cc:1428] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 10:45:23.354035 18346 heartbeater.cc:503] Master 127.10.96.254:44143 was elected leader, sending a full tablet report...
I20260927 10:45:23.354012 18486 heartbeater.cc:503] Master 127.10.96.254:44143 was elected leader, sending a full tablet report...
I20260927 10:45:23.354653 18655 raft_consensus.cc:348] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.354868 18655 raft_consensus.cc:374] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.354921 18655 raft_consensus.cc:729] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd427f690ff4f49b5f4c9f6af5edfd1, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.355346 18655 consensus_queue.cc:260] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.355926 18626 heartbeater.cc:503] Master 127.10.96.254:44143 was elected leader, sending a full tablet report...
I20260927 10:45:23.356215 18655 ts_tablet_manager.cc:1428] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260927 10:45:23.517603 18627 tablet.cc:2369] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:45:23.556137 18487 tablet.cc:2369] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:45:23.588835 18347 tablet.cc:2369] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:45:23.697347 18660 raft_consensus.cc:482] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:23.697642 18660 raft_consensus.cc:504] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.698859 18660 leader_election.cc:288] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951), 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029)
I20260927 10:45:23.702790 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.699282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:36627 (local address 127.10.96.195:38951)
0927 10:45:23.699433 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.699437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.699540 (+ 103us) server_negotiation.cc:407] Connection header received
0927 10:45:23.699719 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.699723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.699793 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.699885 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.700167 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.700681 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.701803 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.702246 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.702373 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.702425 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.702599 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.702666 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:45:23.702803 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.699161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:38951 (local address 127.10.96.193:36627)
0927 10:45:23.699428 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.699445 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.699560 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.699899 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.699902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.699920 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.700052 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.700058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.700791 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.700803 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.701682 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.701691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.702373 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.702377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.702467 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.702491 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.702580 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.702646 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"spinlock_wait_cycles":19072,"thread_start_us":105,"threads_started":1}
I20260927 10:45:23.703001 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.699690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:45425 (local address 127.10.96.194:34029)
0927 10:45:23.699822 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.699828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.699948 (+ 120us) server_negotiation.cc:407] Connection header received
0927 10:45:23.700109 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.700112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.700161 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.700240 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.700514 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.701002 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.702087 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.702478 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.702632 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.702682 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.702827 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.702900 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:45:23.703011 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.699566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34029 (local address 127.10.96.193:45425)
0927 10:45:23.699853 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.699864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.700004 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.700260 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.700263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.700277 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.700405 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.700410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.701111 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.701117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.701967 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.701975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.702607 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.702610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.702703 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.702733 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.702804 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.702871 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20260927 10:45:23.703660 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c004f3fc7044019308f8cac44b7a0a" candidate_uuid: "137ab6c704184cbebdfadd3bbba9ad81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b52055c774efba6b7f19e68cd133f" is_pre_election: true
I20260927 10:45:23.703670 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c004f3fc7044019308f8cac44b7a0a" candidate_uuid: "137ab6c704184cbebdfadd3bbba9ad81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" is_pre_election: true
I20260927 10:45:23.703953 18440 raft_consensus.cc:2454] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137ab6c704184cbebdfadd3bbba9ad81 in term 0.
I20260927 10:45:23.703980 18580 raft_consensus.cc:2454] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137ab6c704184cbebdfadd3bbba9ad81 in term 0.
I20260927 10:45:23.704434 18231 leader_election.cc:302] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 137ab6c704184cbebdfadd3bbba9ad81; no voters:
I20260927 10:45:23.704699 18660 raft_consensus.cc:2781] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:23.704792 18660 raft_consensus.cc:482] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:23.704838 18660 raft_consensus.cc:3037] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.705886 18660 raft_consensus.cc:504] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.706210 18660 leader_election.cc:288] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [CANDIDATE]: Term 1 election: Requested vote from peers 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951), 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029)
I20260927 10:45:23.706604 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c004f3fc7044019308f8cac44b7a0a" candidate_uuid: "137ab6c704184cbebdfadd3bbba9ad81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b52055c774efba6b7f19e68cd133f"
I20260927 10:45:23.706604 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c004f3fc7044019308f8cac44b7a0a" candidate_uuid: "137ab6c704184cbebdfadd3bbba9ad81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
I20260927 10:45:23.706723 18440 raft_consensus.cc:3037] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.706723 18580 raft_consensus.cc:3037] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.708070 18440 raft_consensus.cc:2454] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137ab6c704184cbebdfadd3bbba9ad81 in term 1.
I20260927 10:45:23.708350 18231 leader_election.cc:302] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 137ab6c704184cbebdfadd3bbba9ad81; no voters:
I20260927 10:45:23.708479 18580 raft_consensus.cc:2454] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137ab6c704184cbebdfadd3bbba9ad81 in term 1.
I20260927 10:45:23.708531 18660 raft_consensus.cc:2781] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:23.708798 18660 raft_consensus.cc:686] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 LEADER]: Becoming Leader. State: Replica: 137ab6c704184cbebdfadd3bbba9ad81, State: Running, Role: LEADER
I20260927 10:45:23.709146 18660 consensus_queue.cc:237] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } }
I20260927 10:45:23.712212 18142 catalog_manager.cc:5495] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193). New cstate: current_term: 1 leader_uuid: "137ab6c704184cbebdfadd3bbba9ad81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:23.780055 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.776423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.10.96.193:34087)
0927 10:45:23.776553 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.776558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.776572 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:23.776734 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.776736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.776791 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.776907 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:23.777159 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.777695 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.778571 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.778792 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.778899 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.779192 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:23.779627 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:23.779758 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.779825 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.779883 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:45:23.786322 18440 raft_consensus.cc:1261] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Refusing update from remote peer 137ab6c704184cbebdfadd3bbba9ad81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:23.786350 18580 raft_consensus.cc:1261] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Refusing update from remote peer 137ab6c704184cbebdfadd3bbba9ad81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:23.787037 18660 consensus_queue.cc:1035] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:23.787225 18665 consensus_queue.cc:1035] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:23.797025 18668 mvcc.cc:205] Tried to move back new op lower bound from 7333912263820754944 to 7333912263515258880. Current Snapshot: MvccSnapshot[applied={T|T < 7333912263820754944 or (T in {7333912263820754944})}]
I20260927 10:45:23.799206 18670 mvcc.cc:205] Tried to move back new op lower bound from 7333912263820754944 to 7333912263515258880. Current Snapshot: MvccSnapshot[applied={T|T < 7333912263820754944}]
I20260927 10:45:23.803315 18142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:23.803488 18142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:23.805814 18142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Sending DeleteTablet for 3 replicas of tablet 08c004f3fc7044019308f8cac44b7a0a
I20260927 10:45:23.806564 18280 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c004f3fc7044019308f8cac44b7a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:23 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260927 10:45:23.806748 18420 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c004f3fc7044019308f8cac44b7a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:23 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54806
I20260927 10:45:23.807175 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet 08c004f3fc7044019308f8cac44b7a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:23 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:50628
I20260927 10:45:23.807655 18678 tablet_replica.cc:330] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: stopping tablet replica
I20260927 10:45:23.807677 18679 tablet_replica.cc:330] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: stopping tablet replica
I20260927 10:45:23.807775 18142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 10:45:23.807986 18678 raft_consensus.cc:2229] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 LEADER]: Raft consensus shutting down.
W20260927 10:45:23.808238 18142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:23.808238 18679 raft_consensus.cc:2229] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:23.808452 18678 raft_consensus.cc:2258] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:23.808573 18679 raft_consensus.cc:2258] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:23.809897 18680 tablet_replica.cc:330] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: stopping tablet replica
I20260927 10:45:23.810119 18679 ts_tablet_manager.cc:1905] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:23.810293 18680 raft_consensus.cc:2229] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:23.810626 18680 raft_consensus.cc:2258] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:23.812044 18680 ts_tablet_manager.cc:1905] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:23.813197 18679 ts_tablet_manager.cc:1918] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:23.813364 18679 log.cc:1196] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/08c004f3fc7044019308f8cac44b7a0a
I20260927 10:45:23.813853 18679 ts_tablet_manager.cc:1939] T 08c004f3fc7044019308f8cac44b7a0a P 124b52055c774efba6b7f19e68cd133f: Deleting consensus metadata
I20260927 10:45:23.814662 18280 tablet_service.cc:1467] Processing CreateTablet for tablet b2edef2a6b09488290f55f0002b711ed (DEFAULT_TABLE table=test-table [id=7560687826be41c5bf92c7a0771bb269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.814932 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2edef2a6b09488290f55f0002b711ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.814997 18680 ts_tablet_manager.cc:1918] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:23.815157 18680 log.cc:1196] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/08c004f3fc7044019308f8cac44b7a0a
I20260927 10:45:23.815577 18680 ts_tablet_manager.cc:1939] T 08c004f3fc7044019308f8cac44b7a0a P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting consensus metadata
I20260927 10:45:23.817236 18654 tablet_bootstrap.cc:492] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Bootstrap starting.
I20260927 10:45:23.817775 18127 catalog_manager.cc:4841] TS 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951): tablet 08c004f3fc7044019308f8cac44b7a0a (table test-table [id=67ceb66ddd9a45889e115478b3c4715b]) successfully deleted
I20260927 10:45:23.817927 18127 catalog_manager.cc:4841] TS 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029): tablet 08c004f3fc7044019308f8cac44b7a0a (table test-table [id=67ceb66ddd9a45889e115478b3c4715b]) successfully deleted
I20260927 10:45:23.818107 18420 tablet_service.cc:1467] Processing CreateTablet for tablet b2edef2a6b09488290f55f0002b711ed (DEFAULT_TABLE table=test-table [id=7560687826be41c5bf92c7a0771bb269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.818377 18654 tablet_bootstrap.cc:654] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.818375 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2edef2a6b09488290f55f0002b711ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.818894 18678 ts_tablet_manager.cc:1905] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:23.819321 18560 tablet_service.cc:1467] Processing CreateTablet for tablet b2edef2a6b09488290f55f0002b711ed (DEFAULT_TABLE table=test-table [id=7560687826be41c5bf92c7a0771bb269]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:23.819582 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2edef2a6b09488290f55f0002b711ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:23.821064 18656 tablet_bootstrap.cc:492] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Bootstrap starting.
I20260927 10:45:23.821957 18678 ts_tablet_manager.cc:1918] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:23.822009 18655 tablet_bootstrap.cc:492] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Bootstrap starting.
I20260927 10:45:23.822042 18678 log.cc:1196] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/08c004f3fc7044019308f8cac44b7a0a
I20260927 10:45:23.822108 18656 tablet_bootstrap.cc:654] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.822342 18678 ts_tablet_manager.cc:1939] T 08c004f3fc7044019308f8cac44b7a0a P 137ab6c704184cbebdfadd3bbba9ad81: Deleting consensus metadata
I20260927 10:45:23.823057 18655 tablet_bootstrap.cc:654] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:23.823661 18129 catalog_manager.cc:4841] TS 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087): tablet 08c004f3fc7044019308f8cac44b7a0a (table test-table [id=67ceb66ddd9a45889e115478b3c4715b]) successfully deleted
I20260927 10:45:23.824476 18654 tablet_bootstrap.cc:492] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: No bootstrap required, opened a new log
I20260927 10:45:23.824554 18654 ts_tablet_manager.cc:1397] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 10:45:23.824998 18654 raft_consensus.cc:348] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.825137 18654 raft_consensus.cc:374] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.825184 18654 raft_consensus.cc:729] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137ab6c704184cbebdfadd3bbba9ad81, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.825314 18654 consensus_queue.cc:260] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.825614 18654 ts_tablet_manager.cc:1428] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:23.826388 18655 tablet_bootstrap.cc:492] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: No bootstrap required, opened a new log
I20260927 10:45:23.826485 18656 tablet_bootstrap.cc:492] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: No bootstrap required, opened a new log
I20260927 10:45:23.826499 18655 ts_tablet_manager.cc:1397] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 10:45:23.826573 18656 ts_tablet_manager.cc:1397] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 10:45:23.826921 18655 raft_consensus.cc:348] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.827046 18655 raft_consensus.cc:374] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.827095 18655 raft_consensus.cc:729] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd427f690ff4f49b5f4c9f6af5edfd1, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.827243 18655 consensus_queue.cc:260] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.827263 18656 raft_consensus.cc:348] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.827378 18656 raft_consensus.cc:374] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:23.827409 18656 raft_consensus.cc:729] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Initialized, Role: FOLLOWER
I20260927 10:45:23.827531 18655 ts_tablet_manager.cc:1428] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 10:45:23.827535 18656 consensus_queue.cc:260] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.827768 18656 ts_tablet_manager.cc:1428] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:23.943250 18661 raft_consensus.cc:482] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:23.943408 18661 raft_consensus.cc:504] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.944725 18661 leader_election.cc:288] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:23.949381 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.945046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:35437 (local address 127.10.96.195:38951)
0927 10:45:23.945173 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.945180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.945625 (+ 445us) server_negotiation.cc:407] Connection header received
0927 10:45:23.945721 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.945724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.945791 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.945865 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.946165 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.946692 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.948206 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.948769 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.948897 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.948952 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.949190 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.949266 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:45:23.949410 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.944976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:38951 (local address 127.10.96.194:35437)
0927 10:45:23.945518 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.945532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.945637 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.945893 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.945896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.945912 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.946042 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.946048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.946809 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.946815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.948068 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.948078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.948918 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.948922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.949023 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.949081 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.949170 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.949241 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":128,"thread_start_us":109,"threads_started":1}
I20260927 10:45:23.949551 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.945323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:56181 (local address 127.10.96.193:34087)
0927 10:45:23.945438 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.945445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.945525 (+ 80us) server_negotiation.cc:407] Connection header received
0927 10:45:23.945721 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.945724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.945772 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.945833 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:23.946254 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.947004 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.948397 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.948795 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.948927 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.948980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:23.949329 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 10:45:23.949407 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:45:23.950063 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.944976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34087 (local address 127.10.96.194:56181)
0927 10:45:23.945430 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.945445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.945576 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.945979 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.945982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.946005 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:23.946126 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.946134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.947126 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.947129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.948262 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.948269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.949026 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.949029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.949176 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.949197 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:45:23.949852 (+ 655us) client_negotiation.cc:240] Negotiation successful
0927 10:45:23.949921 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":58,"thread_start_us":66,"threads_started":1}
I20260927 10:45:23.950026 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2edef2a6b09488290f55f0002b711ed" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" is_pre_election: true
I20260927 10:45:23.950184 18580 raft_consensus.cc:2454] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 0.
I20260927 10:45:23.950557 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2edef2a6b09488290f55f0002b711ed" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81" is_pre_election: true
I20260927 10:45:23.950752 18371 leader_election.cc:302] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:23.950790 18300 raft_consensus.cc:2454] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 0.
I20260927 10:45:23.951018 18661 raft_consensus.cc:2781] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:23.951071 18661 raft_consensus.cc:482] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:23.951107 18661 raft_consensus.cc:3037] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.952454 18661 raft_consensus.cc:504] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.952800 18661 leader_election.cc:288] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 election: Requested vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:23.953179 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2edef2a6b09488290f55f0002b711ed" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
I20260927 10:45:23.953315 18580 raft_consensus.cc:3037] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.954576 18580 raft_consensus.cc:2454] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 1.
I20260927 10:45:23.954849 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2edef2a6b09488290f55f0002b711ed" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81"
I20260927 10:45:23.954888 18371 leader_election.cc:302] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:23.954967 18300 raft_consensus.cc:3037] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:23.955058 18661 raft_consensus.cc:2781] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:23.955251 18661 raft_consensus.cc:686] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 1 LEADER]: Becoming Leader. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Running, Role: LEADER
I20260927 10:45:23.955487 18661 consensus_queue.cc:237] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:23.956022 18300 raft_consensus.cc:2454] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 1.
I20260927 10:45:23.957769 18141 catalog_manager.cc:5495] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f reported cstate change: term changed from 0 to 1, leader changed from <none> to 124b52055c774efba6b7f19e68cd133f (127.10.96.194). New cstate: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } health_report { overall_health: UNKNOWN } } }
Sep 27 10:45:23 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505922, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.254@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:24.003413 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.995214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:44143 (local address 127.0.0.1:33738)
0927 10:45:23.995546 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:23.995559 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:23.995660 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:23.995914 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:23.995916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:23.996288 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:23.996409 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.996415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.997145 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.997147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.997776 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:23.997784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.998180 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.998183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:23.998240 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:23.998781 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:23.998792 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:24.000297 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:24.001955 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:24.001963 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:24.001968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:24.002189 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:24.002460 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:24.002463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:24.002464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:24.002539 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:24.002934 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:24.002939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:24.003036 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:45:24.003126 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:45:24.003241 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":131,"threads_started":1}
I20260927 10:45:24.003438 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:23.995247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.10.96.254:44143)
0927 10:45:23.995502 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:23.995509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:23.995632 (+ 123us) server_negotiation.cc:407] Connection header received
0927 10:45:23.995741 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:23.995743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:23.995783 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:23.995883 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:23.996517 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.997058 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.997885 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:23.998091 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:23.998165 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.000418 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:24.000421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:24.000423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:24.000442 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:24.001862 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:24.002271 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:24.002273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:24.002275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:24.002368 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:24.002641 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:24.002644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:24.002645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:24.002845 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:24.002922 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.003179 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.003299 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
I20260927 10:45:24.007355 18141 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:33738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 10:45:24.008095 18141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:24.012918 18420 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1ee34a516e4c76a5ad814b05d9bb36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7bb3a1c98da435e8eab647fd3f22887]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:45:24.012918 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1ee34a516e4c76a5ad814b05d9bb36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7bb3a1c98da435e8eab647fd3f22887]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:45:24.012969 18560 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1ee34a516e4c76a5ad814b05d9bb36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7bb3a1c98da435e8eab647fd3f22887]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 10:45:24.013309 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1ee34a516e4c76a5ad814b05d9bb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.013334 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1ee34a516e4c76a5ad814b05d9bb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.013350 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1ee34a516e4c76a5ad814b05d9bb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.015880 18655 tablet_bootstrap.cc:492] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Bootstrap starting.
I20260927 10:45:24.016043 18654 tablet_bootstrap.cc:492] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: Bootstrap starting.
I20260927 10:45:24.016048 18656 tablet_bootstrap.cc:492] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: Bootstrap starting.
I20260927 10:45:24.016877 18656 tablet_bootstrap.cc:654] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.016875 18654 tablet_bootstrap.cc:654] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.016889 18655 tablet_bootstrap.cc:654] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.018019 18654 tablet_bootstrap.cc:492] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: No bootstrap required, opened a new log
I20260927 10:45:24.018126 18654 ts_tablet_manager.cc:1397] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:24.018141 18655 tablet_bootstrap.cc:492] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: No bootstrap required, opened a new log
I20260927 10:45:24.018178 18656 tablet_bootstrap.cc:492] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: No bootstrap required, opened a new log
I20260927 10:45:24.018216 18655 ts_tablet_manager.cc:1397] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 10:45:24.018240 18656 ts_tablet_manager.cc:1397] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:24.018563 18654 raft_consensus.cc:348] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.018682 18654 raft_consensus.cc:374] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.018632 18656 raft_consensus.cc:348] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.018632 18655 raft_consensus.cc:348] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.018716 18654 raft_consensus.cc:729] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137ab6c704184cbebdfadd3bbba9ad81, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.018733 18655 raft_consensus.cc:374] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.018733 18656 raft_consensus.cc:374] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.018766 18655 raft_consensus.cc:729] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd427f690ff4f49b5f4c9f6af5edfd1, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.018766 18656 raft_consensus.cc:729] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.018831 18654 consensus_queue.cc:260] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.018886 18656 consensus_queue.cc:260] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.018882 18655 consensus_queue.cc:260] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.019101 18654 ts_tablet_manager.cc:1428] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:24.019120 18655 ts_tablet_manager.cc:1428] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 10:45:24.019119 18656 ts_tablet_manager.cc:1428] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:24.019318 18661 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.019312 18675 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.019428 18661 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.019438 18675 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.019733 18662 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.019856 18662 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.160048 18661 raft_consensus.cc:482] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:24.160193 18661 raft_consensus.cc:504] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.160697 18661 leader_election.cc:288] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:24.161175 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ee34a516e4c76a5ad814b05d9bb36" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81" is_pre_election: true
I20260927 10:45:24.161222 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ee34a516e4c76a5ad814b05d9bb36" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" is_pre_election: true
I20260927 10:45:24.161345 18300 raft_consensus.cc:2454] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 0.
I20260927 10:45:24.161356 18580 raft_consensus.cc:2454] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 0.
I20260927 10:45:24.161737 18371 leader_election.cc:302] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:24.161914 18661 raft_consensus.cc:2781] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:24.161976 18661 raft_consensus.cc:482] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:24.162027 18661 raft_consensus.cc:3037] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.162979 18661 raft_consensus.cc:504] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.163336 18661 leader_election.cc:288] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 election: Requested vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:24.163655 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ee34a516e4c76a5ad814b05d9bb36" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
I20260927 10:45:24.163688 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1ee34a516e4c76a5ad814b05d9bb36" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81"
I20260927 10:45:24.163757 18580 raft_consensus.cc:3037] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.163885 18300 raft_consensus.cc:3037] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.164686 18580 raft_consensus.cc:2454] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 1.
I20260927 10:45:24.164878 18300 raft_consensus.cc:2454] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 1.
I20260927 10:45:24.164952 18371 leader_election.cc:302] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:24.165148 18661 raft_consensus.cc:2781] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:24.165251 18661 raft_consensus.cc:686] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [term 1 LEADER]: Becoming Leader. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Running, Role: LEADER
I20260927 10:45:24.165400 18661 consensus_queue.cc:237] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.166143 18686 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: TxnStatusTablet state changed. Reason: New leader 124b52055c774efba6b7f19e68cd133f. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.166255 18686 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:45:24.166554 18699 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 10:45:24.166707 18699 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 10:45:24.167683 18143 catalog_manager.cc:5495] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f reported cstate change: term changed from 0 to 1, leader changed from <none> to 124b52055c774efba6b7f19e68cd133f (127.10.96.194). New cstate: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:24.212932 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.208661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.10.96.194:34029)
0927 10:45:24.209060 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.209064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.209075 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:45:24.209145 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.209147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.209194 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.209333 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:24.209634 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.210167 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.211407 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.211648 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.211740 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.212078 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.212546 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.212638 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.212745 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.212794 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":96,"threads_started":1}
I20260927 10:45:24.212955 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.208493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34029 (local address 127.0.0.1:39380)
0927 10:45:24.208725 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:24.208745 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:24.208870 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:24.209362 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:24.209365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:24.209372 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:45:24.209529 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.209535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.210301 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.210304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.211275 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.211285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.211759 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.211762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.211866 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.211917 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.212639 (+ 722us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.212653 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 10:45:24.212724 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:45:24.212807 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 10:45:24.216035 18580 raft_consensus.cc:1261] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Refusing update from remote peer 124b52055c774efba6b7f19e68cd133f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.216234 18300 raft_consensus.cc:1261] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Refusing update from remote peer 124b52055c774efba6b7f19e68cd133f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.216530 18686 consensus_queue.cc:1035] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.216809 18661 consensus_queue.cc:1035] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.218465 18662 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: TxnStatusTablet state changed. Reason: New leader 124b52055c774efba6b7f19e68cd133f. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.218585 18662 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.218523 18675 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: TxnStatusTablet state changed. Reason: New leader 124b52055c774efba6b7f19e68cd133f. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.218622 18675 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.220618 18669 mvcc.cc:205] Tried to move back new op lower bound from 7333912265583538176 to 7333912265383059456. Current Snapshot: MvccSnapshot[applied={T|T < 7333912265583538176}]
I20260927 10:45:24.220724 18661 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.220842 18661 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:45:24.221632 18686 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.221729 18686 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 124b52055c774efba6b7f19e68cd133f: This TxnStatusTablet replica's current role is: LEADER
I20260927 10:45:24.222851 18662 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.222972 18662 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 8fd427f690ff4f49b5f4c9f6af5edfd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.223635 18675 tablet_replica.cc:439] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } } }
I20260927 10:45:24.223762 18675 tablet_replica.cc:442] T 8e1ee34a516e4c76a5ad814b05d9bb36 P 137ab6c704184cbebdfadd3bbba9ad81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 10:45:24.230461 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.227137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.10.96.194:34029)
0927 10:45:24.227428 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.227431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.227440 (+ 9us) server_negotiation.cc:407] Connection header received
0927 10:45:24.227471 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.227473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.227516 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.227571 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:24.227835 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.228346 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.229520 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.229748 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.229830 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.230082 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.230147 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.230201 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.230327 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.230367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260927 10:45:24.241400 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.236873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:35505 (local address 127.10.96.194:34029)
0927 10:45:24.236982 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.236986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.237180 (+ 194us) server_negotiation.cc:407] Connection header received
0927 10:45:24.237305 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.237307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.237342 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.237399 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:24.237666 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.238140 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.240171 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.240495 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.240584 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.240913 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.240974 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.241029 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.241156 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.241215 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
I20260927 10:45:24.241434 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.236813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34029 (local address 127.10.96.194:35505)
0927 10:45:24.237099 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:24.237114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:24.237224 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:24.237425 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:24.237429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:24.237435 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 10:45:24.237576 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.237582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.238232 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.238235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.240024 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.240035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.240605 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.240609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.240737 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.240792 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.241053 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.241059 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:45:24.241134 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:45:24.241211 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
I20260927 10:45:24.248579 18580 raft_consensus.cc:1261] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Refusing update from remote peer 124b52055c774efba6b7f19e68cd133f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.248582 18300 raft_consensus.cc:1261] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Refusing update from remote peer 124b52055c774efba6b7f19e68cd133f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.248994 18686 consensus_queue.cc:1035] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.249250 18686 consensus_queue.cc:1035] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.292472 18141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:24.292618 18141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:24.294775 18141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Sending DeleteTablet for 3 replicas of tablet b2edef2a6b09488290f55f0002b711ed
I20260927 10:45:24.295405 18419 tablet_service.cc:1514] Processing DeleteTablet for tablet b2edef2a6b09488290f55f0002b711ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54806
I20260927 10:45:24.295433 18280 tablet_service.cc:1514] Processing DeleteTablet for tablet b2edef2a6b09488290f55f0002b711ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260927 10:45:24.295611 18679 tablet_replica.cc:330] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: stopping tablet replica
I20260927 10:45:24.295639 18678 tablet_replica.cc:330] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: stopping tablet replica
I20260927 10:45:24.295715 18679 raft_consensus.cc:2229] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:24.295787 18678 raft_consensus.cc:2229] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:24.295838 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet b2edef2a6b09488290f55f0002b711ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:50628
I20260927 10:45:24.295912 18678 raft_consensus.cc:2258] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.296001 18680 tablet_replica.cc:330] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: stopping tablet replica
I20260927 10:45:24.296044 18679 raft_consensus.cc:2258] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.296109 18680 raft_consensus.cc:2229] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:24.296216 18680 raft_consensus.cc:2258] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.297016 18679 ts_tablet_manager.cc:1905] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.297016 18678 ts_tablet_manager.cc:1905] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.299842 18678 ts_tablet_manager.cc:1918] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:45:24.299842 18679 ts_tablet_manager.cc:1918] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:45:24.299916 18679 log.cc:1196] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/b2edef2a6b09488290f55f0002b711ed
I20260927 10:45:24.299916 18678 log.cc:1196] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/b2edef2a6b09488290f55f0002b711ed
I20260927 10:45:24.300172 18679 ts_tablet_manager.cc:1939] T b2edef2a6b09488290f55f0002b711ed P 124b52055c774efba6b7f19e68cd133f: Deleting consensus metadata
I20260927 10:45:24.300172 18678 ts_tablet_manager.cc:1939] T b2edef2a6b09488290f55f0002b711ed P 137ab6c704184cbebdfadd3bbba9ad81: Deleting consensus metadata
I20260927 10:45:24.300836 18127 catalog_manager.cc:4841] TS 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029): tablet b2edef2a6b09488290f55f0002b711ed (table test-table [id=7560687826be41c5bf92c7a0771bb269]) successfully deleted
I20260927 10:45:24.301146 18129 catalog_manager.cc:4841] TS 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087): tablet b2edef2a6b09488290f55f0002b711ed (table test-table [id=7560687826be41c5bf92c7a0771bb269]) successfully deleted
I20260927 10:45:24.301422 18680 ts_tablet_manager.cc:1905] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.303824 18680 ts_tablet_manager.cc:1918] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:45:24.303879 18680 log.cc:1196] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/b2edef2a6b09488290f55f0002b711ed
I20260927 10:45:24.304059 18680 ts_tablet_manager.cc:1939] T b2edef2a6b09488290f55f0002b711ed P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting consensus metadata
I20260927 10:45:24.304819 18127 catalog_manager.cc:4841] TS 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951): tablet b2edef2a6b09488290f55f0002b711ed (table test-table [id=7560687826be41c5bf92c7a0771bb269]) successfully deleted
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:45:24.325256 18693 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:24.318060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.10.96.254:44143)
0927 10:45:24.318200 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.318207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.318363 (+ 156us) server_negotiation.cc:407] Connection header received
0927 10:45:24.318506 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.318509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.318551 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.318624 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:24.319532 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.320048 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.321231 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.321576 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.321655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.325120 (+ 3465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33748: BlockingRecv error: recv got EOF from 127.0.0.1:33748 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM
I20260927 10:45:24.335595 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.326858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.10.96.254:44143)
0927 10:45:24.326964 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.326969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.327097 (+ 128us) server_negotiation.cc:407] Connection header received
0927 10:45:24.327188 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.327191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.327238 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.327314 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:24.327848 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.328608 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.329359 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.329760 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.329869 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.332176 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:24.332180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:24.332183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:24.332210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:24.333902 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:24.334282 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:24.334285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:24.334287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:24.334384 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:24.334692 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:24.334703 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:24.334705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:24.334964 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:24.335114 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.335310 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.335435 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:45:24.338418 18141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:24.338886 18141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:24.343674 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 2a18c51fce4940f6881a10e23587cb1e (DEFAULT_TABLE table=test-table [id=719d1bf50bca42939b12f8f7476da4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:24.343783 18419 tablet_service.cc:1467] Processing CreateTablet for tablet 2a18c51fce4940f6881a10e23587cb1e (DEFAULT_TABLE table=test-table [id=719d1bf50bca42939b12f8f7476da4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:24.343914 18560 tablet_service.cc:1467] Processing CreateTablet for tablet 2a18c51fce4940f6881a10e23587cb1e (DEFAULT_TABLE table=test-table [id=719d1bf50bca42939b12f8f7476da4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:24.343981 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a18c51fce4940f6881a10e23587cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.344074 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a18c51fce4940f6881a10e23587cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.344157 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a18c51fce4940f6881a10e23587cb1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.346495 18656 tablet_bootstrap.cc:492] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Bootstrap starting.
I20260927 10:45:24.346647 18654 tablet_bootstrap.cc:492] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Bootstrap starting.
I20260927 10:45:24.347229 18656 tablet_bootstrap.cc:654] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.347464 18654 tablet_bootstrap.cc:654] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.347725 18655 tablet_bootstrap.cc:492] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Bootstrap starting.
I20260927 10:45:24.348500 18655 tablet_bootstrap.cc:654] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.348603 18654 tablet_bootstrap.cc:492] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: No bootstrap required, opened a new log
I20260927 10:45:24.348678 18654 ts_tablet_manager.cc:1397] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:24.349085 18654 raft_consensus.cc:348] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.349196 18654 raft_consensus.cc:374] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.349229 18654 raft_consensus.cc:729] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137ab6c704184cbebdfadd3bbba9ad81, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.349350 18654 consensus_queue.cc:260] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.349647 18654 ts_tablet_manager.cc:1428] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 10:45:24.350304 18656 tablet_bootstrap.cc:492] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: No bootstrap required, opened a new log
I20260927 10:45:24.350391 18656 ts_tablet_manager.cc:1397] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 10:45:24.350747 18656 raft_consensus.cc:348] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.350831 18656 raft_consensus.cc:374] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.350852 18656 raft_consensus.cc:729] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.350941 18656 consensus_queue.cc:260] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.351147 18656 ts_tablet_manager.cc:1428] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:45:24.352098 18655 tablet_bootstrap.cc:492] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: No bootstrap required, opened a new log
I20260927 10:45:24.352195 18655 ts_tablet_manager.cc:1397] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260927 10:45:24.352596 18655 raft_consensus.cc:348] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.352712 18655 raft_consensus.cc:374] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.352742 18655 raft_consensus.cc:729] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd427f690ff4f49b5f4c9f6af5edfd1, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.352852 18655 consensus_queue.cc:260] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.353098 18655 ts_tablet_manager.cc:1428] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:45:24.430682 18662 raft_consensus.cc:482] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:24.430847 18662 raft_consensus.cc:504] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.431927 18662 leader_election.cc:288] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029)
I20260927 10:45:24.437237 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.432304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:55425 (local address 127.10.96.193:34087)
0927 10:45:24.432424 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.432428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.432780 (+ 352us) server_negotiation.cc:407] Connection header received
0927 10:45:24.432952 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.432956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.433005 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.433077 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:24.433424 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.433977 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.435813 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.436435 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.436554 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.436603 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.436919 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.437034 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:24.437775 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.432146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34087 (local address 127.10.96.195:55425)
0927 10:45:24.432671 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:24.432686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:24.432826 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:24.433138 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:24.433140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:24.433153 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:24.433312 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.433318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.434099 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.434102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.435683 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.435692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.436567 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.436572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.436726 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.436753 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:45:24.437493 (+ 740us) client_negotiation.cc:240] Negotiation successful
0927 10:45:24.437591 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":380,"thread_start_us":91,"threads_started":1}
I20260927 10:45:24.437949 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.432475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:38407 (local address 127.10.96.194:34029)
0927 10:45:24.432651 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.432656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.432672 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:45:24.432857 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.432862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.432920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.433017 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:24.433312 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.434094 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.436659 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.437259 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.437371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.437416 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.437758 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.437831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 10:45:24.438287 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.432143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:34029 (local address 127.10.96.195:38407)
0927 10:45:24.432570 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:24.432584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:24.432681 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:24.433027 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:24.433029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:24.433043 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:24.433185 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.433193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.434195 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.434198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.435682 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:24.435692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.437502 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.437505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:24.437609 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.437632 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:45:24.438077 (+ 445us) client_negotiation.cc:240] Negotiation successful
0927 10:45:24.438149 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":81,"thread_start_us":96,"threads_started":1}
I20260927 10:45:24.438261 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a18c51fce4940f6881a10e23587cb1e" candidate_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81" is_pre_election: true
I20260927 10:45:24.438580 18300 raft_consensus.cc:2454] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fd427f690ff4f49b5f4c9f6af5edfd1 in term 0.
I20260927 10:45:24.438767 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a18c51fce4940f6881a10e23587cb1e" candidate_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b52055c774efba6b7f19e68cd133f" is_pre_election: true
I20260927 10:45:24.438907 18440 raft_consensus.cc:2454] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fd427f690ff4f49b5f4c9f6af5edfd1 in term 0.
I20260927 10:45:24.438966 18513 leader_election.cc:302] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137ab6c704184cbebdfadd3bbba9ad81, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:24.439296 18662 raft_consensus.cc:2781] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:24.439352 18662 raft_consensus.cc:482] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:24.439388 18662 raft_consensus.cc:3037] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.440634 18662 raft_consensus.cc:504] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.440999 18662 leader_election.cc:288] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [CANDIDATE]: Term 1 election: Requested vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029)
I20260927 10:45:24.441283 18440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a18c51fce4940f6881a10e23587cb1e" candidate_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124b52055c774efba6b7f19e68cd133f"
I20260927 10:45:24.441318 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a18c51fce4940f6881a10e23587cb1e" candidate_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81"
I20260927 10:45:24.441421 18300 raft_consensus.cc:3037] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.441432 18440 raft_consensus.cc:3037] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.442574 18440 raft_consensus.cc:2454] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fd427f690ff4f49b5f4c9f6af5edfd1 in term 1.
I20260927 10:45:24.442574 18300 raft_consensus.cc:2454] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fd427f690ff4f49b5f4c9f6af5edfd1 in term 1.
I20260927 10:45:24.442855 18511 leader_election.cc:302] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:24.443132 18662 raft_consensus.cc:2781] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:24.443348 18662 raft_consensus.cc:686] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 LEADER]: Becoming Leader. State: Replica: 8fd427f690ff4f49b5f4c9f6af5edfd1, State: Running, Role: LEADER
I20260927 10:45:24.443598 18662 consensus_queue.cc:237] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.446208 18142 catalog_manager.cc:5495] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195). New cstate: current_term: 1 leader_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } health_report { overall_health: HEALTHY } } }
I20260927 10:45:24.456284 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.452502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.10.96.195:38951)
0927 10:45:24.452733 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.452737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.452839 (+ 102us) server_negotiation.cc:407] Connection header received
0927 10:45:24.453029 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.453031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.453101 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.453218 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:24.453504 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.454021 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.454861 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.455088 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.455194 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.455472 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.455904 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.456031 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.456089 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.456137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260927 10:45:24.458738 18440 raft_consensus.cc:1261] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Refusing update from remote peer 8fd427f690ff4f49b5f4c9f6af5edfd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.458810 18300 raft_consensus.cc:1261] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Refusing update from remote peer 8fd427f690ff4f49b5f4c9f6af5edfd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.459280 18729 consensus_queue.cc:1035] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.459517 18662 consensus_queue.cc:1035] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.473538 18141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:24.473706 18141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:24.476090 18141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Sending DeleteTablet for 3 replicas of tablet 2a18c51fce4940f6881a10e23587cb1e
I20260927 10:45:24.476797 18419 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a18c51fce4940f6881a10e23587cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54806
I20260927 10:45:24.476974 18679 tablet_replica.cc:330] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: stopping tablet replica
I20260927 10:45:24.477099 18679 raft_consensus.cc:2229] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:24.477245 18679 raft_consensus.cc:2258] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.477744 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a18c51fce4940f6881a10e23587cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:50628
I20260927 10:45:24.477916 18680 tablet_replica.cc:330] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: stopping tablet replica
I20260927 10:45:24.478019 18680 raft_consensus.cc:2229] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:24.478308 18680 raft_consensus.cc:2258] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.479096 18680 ts_tablet_manager.cc:1905] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.481781 18680 ts_tablet_manager.cc:1918] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:24.481917 18680 log.cc:1196] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/2a18c51fce4940f6881a10e23587cb1e
I20260927 10:45:24.482003 18280 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a18c51fce4940f6881a10e23587cb1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260927 10:45:24.482220 18678 tablet_replica.cc:330] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: stopping tablet replica
I20260927 10:45:24.482349 18678 raft_consensus.cc:2229] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:24.482473 18678 raft_consensus.cc:2258] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.482883 18679 ts_tablet_manager.cc:1905] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.482920 18680 ts_tablet_manager.cc:1939] T 2a18c51fce4940f6881a10e23587cb1e P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting consensus metadata
I20260927 10:45:24.483651 18127 catalog_manager.cc:4841] TS 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951): tablet 2a18c51fce4940f6881a10e23587cb1e (table test-table [id=719d1bf50bca42939b12f8f7476da4ac]) successfully deleted
I20260927 10:45:24.484356 18678 ts_tablet_manager.cc:1905] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.485347 18679 ts_tablet_manager.cc:1918] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:24.485410 18679 log.cc:1196] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/2a18c51fce4940f6881a10e23587cb1e
I20260927 10:45:24.485704 18679 ts_tablet_manager.cc:1939] T 2a18c51fce4940f6881a10e23587cb1e P 124b52055c774efba6b7f19e68cd133f: Deleting consensus metadata
I20260927 10:45:24.486655 18678 ts_tablet_manager.cc:1918] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:24.486719 18678 log.cc:1196] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/2a18c51fce4940f6881a10e23587cb1e
I20260927 10:45:24.487036 18127 catalog_manager.cc:4841] TS 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029): tablet 2a18c51fce4940f6881a10e23587cb1e (table test-table [id=719d1bf50bca42939b12f8f7476da4ac]) successfully deleted
I20260927 10:45:24.486954 18678 ts_tablet_manager.cc:1939] T 2a18c51fce4940f6881a10e23587cb1e P 137ab6c704184cbebdfadd3bbba9ad81: Deleting consensus metadata
I20260927 10:45:24.487223 18141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:24.487674 18141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:24.488118 18129 catalog_manager.cc:4841] TS 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087): tablet 2a18c51fce4940f6881a10e23587cb1e (table test-table [id=719d1bf50bca42939b12f8f7476da4ac]) successfully deleted
I20260927 10:45:24.492202 18280 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7ac7fd22794ea49ce33d014a277b11 (DEFAULT_TABLE table=test-table [id=5e1443c90f894bdb8f64b605e878f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:24.492462 18280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7ac7fd22794ea49ce33d014a277b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.492471 18419 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7ac7fd22794ea49ce33d014a277b11 (DEFAULT_TABLE table=test-table [id=5e1443c90f894bdb8f64b605e878f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:24.492719 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7ac7fd22794ea49ce33d014a277b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.494731 18654 tablet_bootstrap.cc:492] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: Bootstrap starting.
I20260927 10:45:24.495071 18656 tablet_bootstrap.cc:492] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Bootstrap starting.
I20260927 10:45:24.495482 18654 tablet_bootstrap.cc:654] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.495791 18656 tablet_bootstrap.cc:654] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.496609 18560 tablet_service.cc:1467] Processing CreateTablet for tablet 8e7ac7fd22794ea49ce33d014a277b11 (DEFAULT_TABLE table=test-table [id=5e1443c90f894bdb8f64b605e878f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:24.496853 18560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e7ac7fd22794ea49ce33d014a277b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:24.498051 18654 tablet_bootstrap.cc:492] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: No bootstrap required, opened a new log
I20260927 10:45:24.498127 18654 ts_tablet_manager.cc:1397] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 10:45:24.498494 18654 raft_consensus.cc:348] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.498596 18654 raft_consensus.cc:374] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.498636 18654 raft_consensus.cc:729] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137ab6c704184cbebdfadd3bbba9ad81, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.498787 18654 consensus_queue.cc:260] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.499042 18654 ts_tablet_manager.cc:1428] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:45:24.499058 18655 tablet_bootstrap.cc:492] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Bootstrap starting.
I20260927 10:45:24.499890 18655 tablet_bootstrap.cc:654] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:24.500969 18656 tablet_bootstrap.cc:492] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: No bootstrap required, opened a new log
I20260927 10:45:24.501051 18656 ts_tablet_manager.cc:1397] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 10:45:24.501425 18656 raft_consensus.cc:348] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.501582 18656 raft_consensus.cc:374] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.501616 18656 raft_consensus.cc:729] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.501719 18656 consensus_queue.cc:260] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.501952 18656 ts_tablet_manager.cc:1428] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:45:24.502573 18655 tablet_bootstrap.cc:492] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: No bootstrap required, opened a new log
I20260927 10:45:24.502655 18655 ts_tablet_manager.cc:1397] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 10:45:24.503073 18655 raft_consensus.cc:348] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.503168 18655 raft_consensus.cc:374] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:24.503196 18655 raft_consensus.cc:729] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd427f690ff4f49b5f4c9f6af5edfd1, State: Initialized, Role: FOLLOWER
I20260927 10:45:24.503315 18655 consensus_queue.cc:260] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.503587 18655 ts_tablet_manager.cc:1428] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:45:24.630448 18710 raft_consensus.cc:482] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:24.630618 18710 raft_consensus.cc:504] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.631177 18710 leader_election.cc:288] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:24.631682 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7ac7fd22794ea49ce33d014a277b11" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81" is_pre_election: true
I20260927 10:45:24.631809 18579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7ac7fd22794ea49ce33d014a277b11" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" is_pre_election: true
I20260927 10:45:24.631889 18300 raft_consensus.cc:2454] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 0.
I20260927 10:45:24.631955 18579 raft_consensus.cc:2454] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 0.
I20260927 10:45:24.632232 18373 leader_election.cc:302] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 137ab6c704184cbebdfadd3bbba9ad81; no voters:
I20260927 10:45:24.632470 18710 raft_consensus.cc:2781] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:24.632541 18710 raft_consensus.cc:482] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:24.632601 18710 raft_consensus.cc:3037] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.633797 18710 raft_consensus.cc:504] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.634071 18710 leader_election.cc:288] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 election: Requested vote from peers 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087), 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951)
I20260927 10:45:24.634400 18579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7ac7fd22794ea49ce33d014a277b11" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1"
I20260927 10:45:24.634418 18300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e7ac7fd22794ea49ce33d014a277b11" candidate_uuid: "124b52055c774efba6b7f19e68cd133f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137ab6c704184cbebdfadd3bbba9ad81"
I20260927 10:45:24.634511 18579 raft_consensus.cc:3037] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.634550 18300 raft_consensus.cc:3037] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:24.635396 18579 raft_consensus.cc:2454] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 1.
I20260927 10:45:24.635510 18300 raft_consensus.cc:2454] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124b52055c774efba6b7f19e68cd133f in term 1.
I20260927 10:45:24.635666 18371 leader_election.cc:302] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124b52055c774efba6b7f19e68cd133f, 8fd427f690ff4f49b5f4c9f6af5edfd1; no voters:
I20260927 10:45:24.635854 18710 raft_consensus.cc:2781] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:24.635968 18710 raft_consensus.cc:686] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 1 LEADER]: Becoming Leader. State: Replica: 124b52055c774efba6b7f19e68cd133f, State: Running, Role: LEADER
I20260927 10:45:24.636140 18710 consensus_queue.cc:237] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } }
I20260927 10:45:24.637852 18142 catalog_manager.cc:5495] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f reported cstate change: term changed from 0 to 1, leader changed from <none> to 124b52055c774efba6b7f19e68cd133f (127.10.96.194). New cstate: current_term: 1 leader_uuid: "124b52055c774efba6b7f19e68cd133f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124b52055c774efba6b7f19e68cd133f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 34029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 } health_report { overall_health: UNKNOWN } } }
I20260927 10:45:24.659226 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.656486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.10.96.194:34029)
0927 10:45:24.656590 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:24.656600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:24.656612 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:24.656697 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:24.656699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:24.656738 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:24.656803 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:24.656995 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.657573 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.658389 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:24.658652 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:24.658722 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:24.658927 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:24.658986 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:24.659038 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:24.659091 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 10:45:24.659129 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:45:24.664553 18300 raft_consensus.cc:1261] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Refusing update from remote peer 124b52055c774efba6b7f19e68cd133f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.664553 18579 raft_consensus.cc:1261] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Refusing update from remote peer 124b52055c774efba6b7f19e68cd133f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:24.664937 18710 consensus_queue.cc:1035] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "137ab6c704184cbebdfadd3bbba9ad81" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.665078 18686 consensus_queue.cc:1035] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd427f690ff4f49b5f4c9f6af5edfd1" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 38951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:24.696671 18143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:24.696813 18143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:24.698688 18143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 778717d896d74aa3b476e2ab08f6626d: Sending DeleteTablet for 3 replicas of tablet 8e7ac7fd22794ea49ce33d014a277b11
I20260927 10:45:24.699184 18280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7ac7fd22794ea49ce33d014a277b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260927 10:45:24.699331 18419 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7ac7fd22794ea49ce33d014a277b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54806
I20260927 10:45:24.699424 18678 tablet_replica.cc:330] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: stopping tablet replica
I20260927 10:45:24.699472 18560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e7ac7fd22794ea49ce33d014a277b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:24 UTC) from {username='oryx', principal='oryx/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:50628
I20260927 10:45:24.699544 18678 raft_consensus.cc:2229] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:24.699590 18679 tablet_replica.cc:330] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: stopping tablet replica
I20260927 10:45:24.699653 18678 raft_consensus.cc:2258] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.699676 18680 tablet_replica.cc:330] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: stopping tablet replica
I20260927 10:45:24.699708 18679 raft_consensus.cc:2229] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:24.699898 18679 raft_consensus.cc:2258] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.699766 18680 raft_consensus.cc:2229] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:24.700026 18680 raft_consensus.cc:2258] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:24.700361 18678 ts_tablet_manager.cc:1905] T 8e7ac7fd22794ea49ce33d014a277b11 P 137ab6c704184cbebdfadd3bbba9ad81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.700575 18679 ts_tablet_manager.cc:1905] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.700734 18680 ts_tablet_manager.cc:1905] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:24.700758 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 18215
I20260927 10:45:24.702680 18679 ts_tablet_manager.cc:1918] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:45:24.702730 18679 log.cc:1196] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/8e7ac7fd22794ea49ce33d014a277b11
I20260927 10:45:24.702912 18679 ts_tablet_manager.cc:1939] T 8e7ac7fd22794ea49ce33d014a277b11 P 124b52055c774efba6b7f19e68cd133f: Deleting consensus metadata
I20260927 10:45:24.703742 18680 ts_tablet_manager.cc:1918] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 10:45:24.703836 18680 log.cc:1196] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/8e7ac7fd22794ea49ce33d014a277b11
I20260927 10:45:24.703974 18127 catalog_manager.cc:4841] TS 124b52055c774efba6b7f19e68cd133f (127.10.96.194:34029): tablet 8e7ac7fd22794ea49ce33d014a277b11 (table test-table [id=5e1443c90f894bdb8f64b605e878f038]) successfully deleted
I20260927 10:45:24.704103 18680 ts_tablet_manager.cc:1939] T 8e7ac7fd22794ea49ce33d014a277b11 P 8fd427f690ff4f49b5f4c9f6af5edfd1: Deleting consensus metadata
I20260927 10:45:24.704715 18127 catalog_manager.cc:4841] TS 8fd427f690ff4f49b5f4c9f6af5edfd1 (127.10.96.195:38951): tablet 8e7ac7fd22794ea49ce33d014a277b11 (table test-table [id=5e1443c90f894bdb8f64b605e878f038]) successfully deleted
W20260927 10:45:24.710685 18129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:34087 (error 108)
I20260927 10:45:24.710963 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 18355
I20260927 10:45:24.712515 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:24.711988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34087 (local address 127.0.0.1:45650)
0927 10:45:24.712302 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:24.712386 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:34087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":57,"threads_started":1}
W20260927 10:45:24.712623 18129 catalog_manager.cc:4584] TS 137ab6c704184cbebdfadd3bbba9ad81 (127.10.96.193:34087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e7ac7fd22794ea49ce33d014a277b11: Network error: Client connection negotiation failed: client connection to 127.10.96.193:34087: connect: Connection refused (error 111)
I20260927 10:45:24.718052 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 18495
I20260927 10:45:24.724738 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 18111
2026-09-27T10:45:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3467 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18748](info): set up 2 sockets
Sep 27 10:45:24 dist-test-slave-wmg0 krb5kdc[18748](info): commencing operation
krb5kdc: starting...
W20260927 10:45:25.766681 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:25 dist-test-slave-wmg0 krb5kdc[18748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:25.886430 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37409
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:25.994335 18761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:25.994588 18761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:25.994627 18761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:25.998497 18761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:25.998545 18761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:25.998564 18761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:25.998581 18761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:25.998596 18761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:26.003661 18761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:37409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:26.004628 18761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:26.005601 18761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:26.014377 18767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:26.014473 18766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:26.014683 18761 server_base.cc:1034] running on GCE node
W20260927 10:45:26.014475 18769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:26.015245 18761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:26.016399 18761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:26.017534 18761 hybrid_clock.cc:648] HybridClock initialized: now 1790505926017502 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T10:45:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1297 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:26 dist-test-slave-wmg0 krb5kdc[18774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:26 dist-test-slave-wmg0 krb5kdc[18774](info): set up 2 sockets
Sep 27 10:45:26 dist-test-slave-wmg0 krb5kdc[18774](info): commencing operation
krb5kdc: starting...
W20260927 10:45:27.054338 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:27.176698 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43373
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:27.291059 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:27.291392 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:27.291493 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:27.295625 18787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:27.295713 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:27.295747 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:27.295787 18787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:27.295826 18787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:27.301216 18787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:27.302338 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:27.303354 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:27.311894 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:27.311885 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:27.312047 18787 server_base.cc:1034] running on GCE node
W20260927 10:45:27.311906 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:27.312662 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:27.313776 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:27.314936 18787 hybrid_clock.cc:648] HybridClock initialized: now 1790505927314902 us; error 46 us; skew 500 ppm
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:27.317651 18787 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:27.318485 18787 webserver.cc:460] Webserver started at http://127.10.96.254:32867/ using document root <none> and password file <none>
I20260927 10:45:27.319024 18787 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:27.319084 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:27.319343 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:27.321321 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "b7173cc90dfd818bfbf991cd4b1cd3af"
server_key_iv: "831affa045f92c18b4c0d4b26c023e46"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.321935 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "b7173cc90dfd818bfbf991cd4b1cd3af"
server_key_iv: "831affa045f92c18b4c0d4b26c023e46"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.325418 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:27.327689 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.328881 18787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:27.329016 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "b7173cc90dfd818bfbf991cd4b1cd3af"
server_key_iv: "831affa045f92c18b4c0d4b26c023e46"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.329121 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:27.346807 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:27.349821 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:27.350013 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:27.357390 18787 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43373
I20260927 10:45:27.357409 18854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43373 every 8 connection(s)
I20260927 10:45:27.358397 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:27.361166 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:27.364974 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 18787
I20260927 10:45:27.365077 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:27.365345 10627 external_mini_cluster.cc:1428] Setting key 9d3d16e327d7aba1d1d3bbe76136f985
I20260927 10:45:27.366927 18855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Bootstrap starting.
I20260927 10:45:27.369122 18855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:27.369922 18855 log.cc:824] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:27.371731 18855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: No bootstrap required, opened a new log
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:27.375541 18855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:27.375855 18855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:27.375939 18855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3a0e77e6d94fdaaec6ddb0fd2245c3, State: Initialized, Role: FOLLOWER
I20260927 10:45:27.376425 18855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:27.376573 18855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:27.376641 18855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:27.376747 18855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:27.377730 18855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:27.378072 18855 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3a0e77e6d94fdaaec6ddb0fd2245c3; no voters:
I20260927 10:45:27.378446 18855 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:27.378492 18860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:27.378789 18860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 LEADER]: Becoming Leader. State: Replica: 9e3a0e77e6d94fdaaec6ddb0fd2245c3, State: Running, Role: LEADER
I20260927 10:45:27.379115 18860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:27.379750 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:27.366760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.10.96.254:43373)
0927 10:45:27.367254 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:27.367269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:27.367310 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:45:27.367942 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:27.367959 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:27.368275 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:27.368615 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:27.369499 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.370549 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.371414 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.371925 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.372091 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:27.374662 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:27.374667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:27.374679 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:27.374705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:27.376508 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:27.377146 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:27.377156 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:27.377160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:27.377277 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:27.377619 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:27.377624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:27.377625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:27.378077 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:27.378286 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:27.378581 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 10:45:27.378919 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":133,"threads_started":1}
I20260927 10:45:27.379834 18855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:27.380330 18860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3a0e77e6d94fdaaec6ddb0fd2245c3. Latest consensus state: current_term: 1 leader_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } } }
I20260927 10:45:27.380445 18860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:27.380648 18860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } } }
I20260927 10:45:27.380744 18860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:27.382263 18865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:27.385394 18865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:27.390826 18865 catalog_manager.cc:1347] Generated new cluster ID: c1179c2eac4b4c58a814034a843b6b88
I20260927 10:45:27.390895 18865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:27.408588 18865 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:27.409608 18865 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:27.413499 18865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Generated new TSK 0
I20260927 10:45:27.414280 18865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:45:27.446864 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43373
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:27.555392 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:27.555629 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:27.555670 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:27.559511 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:27.559557 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:27.559612 18881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:27.564640 18881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:27.565590 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:27.566507 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:27.575618 18886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:27.575686 18887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:27.575709 18881 server_base.cc:1034] running on GCE node
W20260927 10:45:27.575618 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:27.576238 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:27.576781 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:27.577940 18881 hybrid_clock.cc:648] HybridClock initialized: now 1790505927577910 us; error 43 us; skew 500 ppm
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:27.586087 18881 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:45:27.587030 18881 webserver.cc:460] Webserver started at http://127.10.96.193:37619/ using document root <none> and password file <none>
I20260927 10:45:27.587618 18881 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:27.587720 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:27.587955 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:27.590118 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "d5554477a1164c8ab92f695bfb08041a"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "42d34c5066c0ad2c9204208f431d8c16"
server_key_iv: "c155736c4fc3d15183d18fcb1008f5d4"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.590693 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "d5554477a1164c8ab92f695bfb08041a"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "42d34c5066c0ad2c9204208f431d8c16"
server_key_iv: "c155736c4fc3d15183d18fcb1008f5d4"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.594434 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:27.596767 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.598016 18881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:27.598148 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "d5554477a1164c8ab92f695bfb08041a"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "42d34c5066c0ad2c9204208f431d8c16"
server_key_iv: "c155736c4fc3d15183d18fcb1008f5d4"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.598268 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:27.613631 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:27.616802 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:27.617036 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:27.617707 18881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:27.618561 18881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:27.618633 18881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.618700 18881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:27.618750 18881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.628312 18881 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39977
I20260927 10:45:27.628325 19009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39977 every 8 connection(s)
I20260927 10:45:27.629156 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:27.632691 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 18881
I20260927 10:45:27.632845 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:27.633178 10627 external_mini_cluster.cc:1428] Setting key 68f9667a4cea8706b82e0aa56937a63c
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:45:27.643832 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:27.630889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:50451 (local address 127.10.96.254:43373)
0927 10:45:27.631035 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:27.631040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:27.631901 (+ 861us) server_negotiation.cc:407] Connection header received
0927 10:45:27.633486 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:27.633490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:27.633545 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:27.633650 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:27.634952 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.635466 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.637081 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.637321 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.637408 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:27.640261 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:27.640264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:27.640266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:27.640289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:27.641836 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:27.642340 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:27.642345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:27.642347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:27.642429 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:27.642772 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:27.642777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:27.642779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:27.642952 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:27.643060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:27.643541 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 10:45:27.643662 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:45:27.644596 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:27.631339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:50451)
0927 10:45:27.631741 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:27.631782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:27.633222 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:27.633823 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:27.633829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:27.634283 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:27.634804 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:27.634817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.635581 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.635585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:27.636931 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:27.636944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.637533 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.637541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:27.637766 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:27.638400 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:27.638433 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:27.640081 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:27.641963 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:27.641969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:27.641983 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:27.642236 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:27.642529 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:27.642532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:27.642534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:27.642671 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:27.643058 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:27.643065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:27.643310 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 10:45:27.643490 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 10:45:27.643803 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:45:27.645897 19010 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43373
I20260927 10:45:27.646131 19010 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:27.646626 19010 heartbeater.cc:511] Master 127.10.96.254:43373 requested a full tablet report, sending...
I20260927 10:45:27.648080 18819 ts_manager.cc:194] Registered new tserver with Master: d5554477a1164c8ab92f695bfb08041a (127.10.96.193:39977)
I20260927 10:45:27.649236 18819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:50451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:45:27.658437 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43373
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:27.766649 19015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:27.766933 19015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:27.767009 19015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:27.771023 19015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:27.771102 19015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:27.771193 19015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:27.776579 19015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:27.777678 19015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:27.778663 19015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:27.788090 19020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:27.788110 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:27.788112 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:27.788543 19015 server_base.cc:1034] running on GCE node
I20260927 10:45:27.788920 19015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:27.789541 19015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:27.790724 19015 hybrid_clock.cc:648] HybridClock initialized: now 1790505927790708 us; error 26 us; skew 500 ppm
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:27.793692 19015 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:45:27.794524 19015 webserver.cc:460] Webserver started at http://127.10.96.194:40333/ using document root <none> and password file <none>
I20260927 10:45:27.795071 19015 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:27.795130 19015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:27.795352 19015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:27.797310 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "20cd4b52d8d547d19b434f1411be4ceb"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "9278790ac92d746b2e5c72834a6c7a25"
server_key_iv: "83e0425477506c169ece0d349e7e6fb7"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.797950 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "20cd4b52d8d547d19b434f1411be4ceb"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "9278790ac92d746b2e5c72834a6c7a25"
server_key_iv: "83e0425477506c169ece0d349e7e6fb7"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.801589 19015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:27.804018 19030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.805197 19015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:27.805325 19015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "20cd4b52d8d547d19b434f1411be4ceb"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "9278790ac92d746b2e5c72834a6c7a25"
server_key_iv: "83e0425477506c169ece0d349e7e6fb7"
server_key_version: "encryptionkey@0"
I20260927 10:45:27.805428 19015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:27.821844 19015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:27.824846 19015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:27.825038 19015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:27.825616 19015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:27.826397 19015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:27.826465 19015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.826534 19015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:27.826588 19015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:27.835320 19015 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:41563
I20260927 10:45:27.835362 19143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:41563 every 8 connection(s)
I20260927 10:45:27.836156 19015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:27.844625 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19015
I20260927 10:45:27.844754 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:27.845059 10627 external_mini_cluster.cc:1428] Setting key b8525320e3075e41047658a96046500f
Sep 27 10:45:27 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505927, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:27.849362 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:27.837863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:44487 (local address 127.10.96.254:43373)
0927 10:45:27.838008 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:27.838013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:27.838874 (+ 861us) server_negotiation.cc:407] Connection header received
0927 10:45:27.840059 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:27.840062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:27.840107 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:27.840180 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:27.841290 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.841876 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.842948 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.843151 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.843225 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:27.845815 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:27.845819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:27.845821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:27.845848 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:27.847438 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:27.847939 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:27.847943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:27.847945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:27.848025 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:27.848346 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:27.848351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:27.848352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:27.848522 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:27.848625 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:27.849078 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 10:45:27.849187 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:45:27.850131 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:27.838299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:44487)
0927 10:45:27.838725 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:27.838766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:27.839870 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:27.840358 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:27.840364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:27.840702 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:27.841152 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:27.841165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.841979 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.841982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:27.842847 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:27.842855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:27.843289 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:27.843291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:27.843422 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:27.843945 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:27.843960 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:27.845624 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:27.847566 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:27.847571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:27.847585 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:27.847836 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:27.848119 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:27.848122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:27.848124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:27.848252 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:27.848621 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:27.848627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:27.848859 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 10:45:27.849038 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 10:45:27.849328 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260927 10:45:27.851336 19144 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43373
I20260927 10:45:27.851560 19144 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:27.852005 19144 heartbeater.cc:511] Master 127.10.96.254:43373 requested a full tablet report, sending...
I20260927 10:45:27.853077 18819 ts_manager.cc:194] Registered new tserver with Master: 20cd4b52d8d547d19b434f1411be4ceb (127.10.96.194:41563)
I20260927 10:45:27.853921 18819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:44487
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:45:27.870738 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43373
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:27.980268 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:27.980522 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:27.980611 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:27.984935 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:27.984992 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:27.985091 19149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:27.990309 19149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:27.991334 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:27.992337 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:28.001585 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.001663 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.001606 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.001971 19149 server_base.cc:1034] running on GCE node
I20260927 10:45:28.002355 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:28.002991 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:28.004169 19149 hybrid_clock.cc:648] HybridClock initialized: now 1790505928004157 us; error 33 us; skew 500 ppm
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505928, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:28.006949 19149 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:28.007768 19149 webserver.cc:460] Webserver started at http://127.10.96.195:35623/ using document root <none> and password file <none>
I20260927 10:45:28.008330 19149 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:28.008419 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:28.008641 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:28.010790 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "abd9fa5ef1934660a84edae7be99880a"
format_stamp: "Formatted at 2026-09-27 10:45:28 on dist-test-slave-wmg0"
server_key: "4b9f94c913b928bd0d2cfc71bbc7d658"
server_key_iv: "46165ac47e08d74ff026c6c1ea326afb"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.011387 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "abd9fa5ef1934660a84edae7be99880a"
format_stamp: "Formatted at 2026-09-27 10:45:28 on dist-test-slave-wmg0"
server_key: "4b9f94c913b928bd0d2cfc71bbc7d658"
server_key_iv: "46165ac47e08d74ff026c6c1ea326afb"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.015028 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:45:28.017436 19164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.018714 19149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:28.018853 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "abd9fa5ef1934660a84edae7be99880a"
format_stamp: "Formatted at 2026-09-27 10:45:28 on dist-test-slave-wmg0"
server_key: "4b9f94c913b928bd0d2cfc71bbc7d658"
server_key_iv: "46165ac47e08d74ff026c6c1ea326afb"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.018971 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:28.038365 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:28.041713 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:28.041914 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:28.042490 19149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:28.043414 19149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:28.043488 19149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.043557 19149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:28.043591 19149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.051973 19149 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40679
I20260927 10:45:28.052011 19277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40679 every 8 connection(s)
I20260927 10:45:28.052932 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:28.056815 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19149
I20260927 10:45:28.056916 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:28.057222 10627 external_mini_cluster.cc:1428] Setting key 61b5bee3399302972706d65b91edfc72
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505928, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:28.066810 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.054757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:48637 (local address 127.10.96.254:43373)
0927 10:45:28.054908 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.054911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.055789 (+ 878us) server_negotiation.cc:407] Connection header received
0927 10:45:28.057225 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.057230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.057291 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.057388 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.059001 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.059665 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.060800 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.061008 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.061087 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.063558 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:28.063561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:28.063563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:28.063585 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:28.065004 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:28.065446 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:28.065448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:28.065450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:28.065589 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:28.065928 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:28.065930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:28.065931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:28.066094 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:28.066170 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:28.066599 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 10:45:28.066692 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:45:28.067570 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.055156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:48637)
0927 10:45:28.055601 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.055646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.056988 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.057629 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.057638 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.058158 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.058858 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.058879 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.059803 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.059806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.060682 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.060690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.061165 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.061168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.061347 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.061904 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.061919 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.063389 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:28.065105 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:28.065109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:28.065120 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:28.065361 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:28.065687 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:28.065690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:28.065691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:28.065840 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:28.066178 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:28.066183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:28.066402 (+ 219us) client_negotiation.cc:769] Sending connection context
0927 10:45:28.066555 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 10:45:28.066801 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":96,"threads_started":1}
I20260927 10:45:28.068689 19278 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43373
I20260927 10:45:28.068940 19278 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:28.069414 19278 heartbeater.cc:511] Master 127.10.96.254:43373 requested a full tablet report, sending...
I20260927 10:45:28.070533 18819 ts_manager.cc:194] Registered new tserver with Master: abd9fa5ef1934660a84edae7be99880a (127.10.96.195:40679)
I20260927 10:45:28.071130 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:45:28.071236 18819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:48637
WARNING: no policy specified for oryx/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab.
Entry for principal oryx/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab.
WARNING: no policy specified for oryx/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab.
Entry for principal oryx/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab.
WARNING: no policy specified for oryx/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab.
Entry for principal oryx/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab.
I20260927 10:45:28.141327 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 18881
I20260927 10:45:28.146827 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19015
I20260927 10:45:28.151932 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19149
I20260927 10:45:28.156905 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 18787
I20260927 10:45:28.163115 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43373
--webserver_interface=127.10.96.254
--webserver_port=32867
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:28.272696 19287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:28.272991 19287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:28.273072 19287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:28.277091 19287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:28.277181 19287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:28.277241 19287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:28.277305 19287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:28.277345 19287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:28.282613 19287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=32867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:28.283671 19287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:28.284705 19287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:28.292824 19292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.292868 19293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.293068 19287 server_base.cc:1034] running on GCE node
W20260927 10:45:28.293126 19295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.293808 19287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:28.294880 19287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:28.296078 19287 hybrid_clock.cc:648] HybridClock initialized: now 1790505928296042 us; error 46 us; skew 500 ppm
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505928, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:28.298915 19287 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:28.299780 19287 webserver.cc:460] Webserver started at http://127.10.96.254:32867/ using document root <none> and password file <none>
I20260927 10:45:28.300289 19287 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:28.300359 19287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:28.304029 19287 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:28.306083 19302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.307288 19287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:28.307425 19287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "b7173cc90dfd818bfbf991cd4b1cd3af"
server_key_iv: "831affa045f92c18b4c0d4b26c023e46"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.307945 19287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:28.319720 19287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:28.322681 19287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:28.322881 19287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:28.330231 19287 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43373
I20260927 10:45:28.330246 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43373 every 8 connection(s)
I20260927 10:45:28.331296 19287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:28.335383 19355 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:45:28.336988 19355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Bootstrap starting.
I20260927 10:45:28.338979 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19287
I20260927 10:45:28.339418 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:39977
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:43373
--webserver_port=37619
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:45:28.353572 19355 log.cc:824] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:28.360220 19355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:45:28.360605 19355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Bootstrap complete.
I20260927 10:45:28.364814 19355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:28.365232 19355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3a0e77e6d94fdaaec6ddb0fd2245c3, State: Initialized, Role: FOLLOWER
I20260927 10:45:28.365780 19355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:28.365921 19355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:28.366006 19355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:28.366101 19355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:45:28.367946 19355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:28.368263 19355 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3a0e77e6d94fdaaec6ddb0fd2245c3; no voters:
I20260927 10:45:28.368610 19355 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:45:28.368672 19359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:45:28.368858 19359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [term 2 LEADER]: Becoming Leader. State: Replica: 9e3a0e77e6d94fdaaec6ddb0fd2245c3, State: Running, Role: LEADER
I20260927 10:45:28.369210 19359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } }
I20260927 10:45:28.369836 19355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:28.370177 19361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } } }
I20260927 10:45:28.370370 19361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:28.370566 19360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3a0e77e6d94fdaaec6ddb0fd2245c3. Latest consensus state: current_term: 2 leader_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3a0e77e6d94fdaaec6ddb0fd2245c3" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43373 } } }
I20260927 10:45:28.370679 19360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:28.370972 19363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:28.374269 19363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:28.374924 19363 catalog_manager.cc:1259] Loaded cluster ID: c1179c2eac4b4c58a814034a843b6b88
I20260927 10:45:28.374986 19363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:28.377269 19363 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:28.377926 19363 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e3a0e77e6d94fdaaec6ddb0fd2245c3: Loaded TSK: 0
I20260927 10:45:28.379117 19363 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:45:28.451396 19357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:28.451682 19357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:28.451766 19357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:28.455906 19357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:28.455986 19357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:28.456072 19357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:28.461179 19357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.193.keytab
--principal=oryx/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:39977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=37619
--tserver_master_addrs=127.10.96.254:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:28.462323 19357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:28.463299 19357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:28.472783 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.472810 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.472990 19384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.473407 19357 server_base.cc:1034] running on GCE node
I20260927 10:45:28.473850 19357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:28.474483 19357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:28.475660 19357 hybrid_clock.cc:648] HybridClock initialized: now 1790505928475640 us; error 34 us; skew 500 ppm
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505928, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:28.478489 19357 init.cc:377] Logged in from keytab as oryx/127.10.96.193@KRBTEST.COM (short username oryx)
I20260927 10:45:28.479420 19357 webserver.cc:460] Webserver started at http://127.10.96.193:37619/ using document root <none> and password file <none>
I20260927 10:45:28.480007 19357 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:28.480077 19357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:28.483873 19357 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:45:28.486048 19391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.487273 19357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:28.487366 19357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "d5554477a1164c8ab92f695bfb08041a"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "42d34c5066c0ad2c9204208f431d8c16"
server_key_iv: "c155736c4fc3d15183d18fcb1008f5d4"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.487897 19357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:28.502508 19357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:28.505594 19357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:28.505784 19357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:28.506397 19357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:28.507275 19357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:28.507323 19357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.507362 19357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:28.507416 19357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.516419 19357 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39977
I20260927 10:45:28.516453 19504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39977 every 8 connection(s)
I20260927 10:45:28.517416 19357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:28.525655 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19357
I20260927 10:45:28.526121 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:41563
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:43373
--webserver_port=40333
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:28.530294 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.519544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:44311)
0927 10:45:28.520005 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.520044 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.521127 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.522139 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.522145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.522476 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.522909 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.522922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.524118 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.524121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.524956 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.524964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.525807 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.525813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.525953 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.526591 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.526616 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.529361 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
W20260927 10:45:28.530287 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.519372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:44311 (local address 127.10.96.254:43373)
0927 10:45:28.519837 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.519856 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.520153 (+ 297us) server_negotiation.cc:407] Connection header received
0927 10:45:28.521495 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.521506 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.521781 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.522047 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.523055 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.524009 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.525051 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.525634 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.525819 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.529587 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:44311: BlockingRecv error: recv got EOF from 127.10.96.193:44311 (error 108)
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":95,"threads_started":1}
W20260927 10:45:28.530732 19505 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:43373 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.96.254:43373: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:28.536891 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.531175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:42455)
0927 10:45:28.531304 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.531318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.531387 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.531660 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.531662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.531785 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.531866 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.531871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.532963 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.532966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.533556 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.533565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.534216 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.534218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.534313 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.534847 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.534858 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.536720 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 10:45:28.536927 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.531255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:42455 (local address 127.10.96.254:43373)
0927 10:45:28.531393 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.531399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.531446 (+ 47us) server_negotiation.cc:407] Connection header received
0927 10:45:28.531489 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.531493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.531550 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.531634 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.531971 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.532844 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.533665 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.534080 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.534209 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.536840 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:42455: BlockingRecv error: recv got EOF from 127.10.96.193:42455 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:28.542404 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.537461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:48429)
0927 10:45:28.537597 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.537611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.537678 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.537892 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.537894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.538008 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.538072 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.538076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.539047 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.539050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.539557 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.539565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.540100 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.540103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.540161 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.540628 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.540638 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.542320 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 10:45:28.542573 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.537564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:48429 (local address 127.10.96.254:43373)
0927 10:45:28.537697 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.537703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.537717 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:28.537750 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.537753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.537792 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.537872 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.538166 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.538937 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.539657 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.539986 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.540079 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.542468 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:48429: BlockingRecv error: recv got EOF from 127.10.96.193:48429 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:45:28.542790 19505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:45:28.637589 19509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:28.637825 19509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:28.637892 19509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:28.641827 19509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:28.641880 19509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:28.641988 19509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:28.647061 19509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.194.keytab
--principal=oryx/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:41563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=40333
--tserver_master_addrs=127.10.96.254:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:28.648132 19509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:28.649127 19509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:28.658257 19517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.658252 19514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.658396 19509 server_base.cc:1034] running on GCE node
W20260927 10:45:28.658252 19515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.658982 19509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:28.659540 19509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:28.660701 19509 hybrid_clock.cc:648] HybridClock initialized: now 1790505928660677 us; error 39 us; skew 500 ppm
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505928, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:28.663394 19509 init.cc:377] Logged in from keytab as oryx/127.10.96.194@KRBTEST.COM (short username oryx)
I20260927 10:45:28.664294 19509 webserver.cc:460] Webserver started at http://127.10.96.194:40333/ using document root <none> and password file <none>
I20260927 10:45:28.664834 19509 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:28.664935 19509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:28.668577 19509 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:28.670730 19524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.671948 19509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:28.672042 19509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "20cd4b52d8d547d19b434f1411be4ceb"
format_stamp: "Formatted at 2026-09-27 10:45:27 on dist-test-slave-wmg0"
server_key: "9278790ac92d746b2e5c72834a6c7a25"
server_key_iv: "83e0425477506c169ece0d349e7e6fb7"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.672569 19509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:28.684160 19509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:28.687304 19509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:28.687500 19509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:28.688051 19509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:28.688818 19509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:28.688861 19509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.688926 19509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:28.688961 19509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.697553 19509 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:41563
I20260927 10:45:28.697587 19637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:41563 every 8 connection(s)
I20260927 10:45:28.698501 19509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:28.702724 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19509
I20260927 10:45:28.703171 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:40679
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:43373
--webserver_port=35623
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:45:28.710668 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.700203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:49997 (local address 127.10.96.254:43373)
0927 10:45:28.700376 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.700381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.701227 (+ 846us) server_negotiation.cc:407] Connection header received
0927 10:45:28.702528 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.702532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.702601 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.702722 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.703925 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.704706 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.706318 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.706738 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.706871 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.710495 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:49997: BlockingRecv error: recv got EOF from 127.10.96.194:49997 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 10:45:28.711128 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.700667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:49997)
0927 10:45:28.701074 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.701114 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.702293 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.702893 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.702900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.703233 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.703775 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.703788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.704838 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.704845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.706122 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.706131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.706888 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.706891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.707033 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.707630 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.707646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.710264 (+ 2618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":100,"threads_started":1}
W20260927 10:45:28.711761 19638 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:43373 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.96.254:43373: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:28.718142 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.712250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:44757)
0927 10:45:28.712503 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.712515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.712609 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.712875 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.712881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.713014 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.713104 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.713109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.714298 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.714300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.714893 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.714902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.715501 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.715503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.715574 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.716134 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.716156 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.718042 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260927 10:45:28.718327 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.712386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:44757 (local address 127.10.96.254:43373)
0927 10:45:28.712561 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.712568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.712585 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:45:28.712699 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.712704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.712760 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.712861 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.713225 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.714147 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.715010 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.715368 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.715472 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.718200 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:44757: BlockingRecv error: recv got EOF from 127.10.96.194:44757 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:28.723802 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.718845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:55089)
0927 10:45:28.719084 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.719098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.719190 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.719411 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.719413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.719579 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.719647 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.719652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.720349 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.720355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.720875 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.720882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.721330 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.721334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.721411 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.721922 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.721936 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.723665 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170}
W20260927 10:45:28.723884 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.718922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:55089 (local address 127.10.96.254:43373)
0927 10:45:28.719061 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.719065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.719161 (+ 96us) server_negotiation.cc:407] Connection header received
0927 10:45:28.719267 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.719271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.719316 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.719396 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.719757 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.720243 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.721008 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.721223 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.721331 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.723770 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:55089: BlockingRecv error: recv got EOF from 127.10.96.194:55089 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 10:45:28.724246 19638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:45:28.820151 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:28.820451 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:28.820528 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:28.824522 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:28.824615 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:28.824705 19641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:28.830494 19641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/krb5kdc/oryx_127.10.96.195.keytab
--principal=oryx/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:40679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=35623
--tserver_master_addrs=127.10.96.254:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:28.831542 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:28.832582 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:28.841997 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.842017 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:28.842051 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:28.842376 19641 server_base.cc:1034] running on GCE node
I20260927 10:45:28.842808 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:28.843415 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:28.844607 19641 hybrid_clock.cc:648] HybridClock initialized: now 1790505928844594 us; error 31 us; skew 500 ppm
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505928, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:28.847443 19641 init.cc:377] Logged in from keytab as oryx/127.10.96.195@KRBTEST.COM (short username oryx)
I20260927 10:45:28.848387 19641 webserver.cc:460] Webserver started at http://127.10.96.195:35623/ using document root <none> and password file <none>
I20260927 10:45:28.848940 19641 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:28.849044 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:28.852875 19641 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:28.855023 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.856228 19641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:28.856370 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "abd9fa5ef1934660a84edae7be99880a"
format_stamp: "Formatted at 2026-09-27 10:45:28 on dist-test-slave-wmg0"
server_key: "4b9f94c913b928bd0d2cfc71bbc7d658"
server_key_iv: "46165ac47e08d74ff026c6c1ea326afb"
server_key_version: "encryptionkey@0"
I20260927 10:45:28.856909 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:28.868732 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:28.872186 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:28.872598 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:28.873193 19641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:28.874110 19641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:28.874181 19641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.874244 19641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:28.874274 19641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:28.883461 19641 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40679
I20260927 10:45:28.883482 19769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40679 every 8 connection(s)
I20260927 10:45:28.884425 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:28.890278 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19641
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:45:28.895994 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.886215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:48739 (local address 127.10.96.254:43373)
0927 10:45:28.886376 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.886380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.887186 (+ 806us) server_negotiation.cc:407] Connection header received
0927 10:45:28.888413 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.888416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.888461 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.888538 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.889834 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.890386 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.892108 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.892421 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.892535 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.895878 (+ 3343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:48739: BlockingRecv error: recv got EOF from 127.10.96.195:48739 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 10:45:28.896678 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.886620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:48739)
0927 10:45:28.887035 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.887076 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.888202 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.888733 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.888740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.889077 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.889667 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.889684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.890532 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.890535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.891979 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.891989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.892570 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.892574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.892745 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.893355 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.893371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.895783 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
W20260927 10:45:28.897289 19770 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:43373 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.96.254:43373: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
I20260927 10:45:28.899999 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.891305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.10.96.254:43373)
0927 10:45:28.891541 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.891546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.891567 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:45:28.891600 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.891602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.891637 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.891739 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.892556 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.893075 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.893999 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.894221 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.894327 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.895766 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:28.895782 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:28.895797 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:28.895823 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:28.897919 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:28.898372 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:28.898378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:28.898384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:28.898507 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:28.898792 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:28.898795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:28.898797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:28.899125 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:28.899322 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:28.899610 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 10:45:28.899863 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:45:28.902545 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.897807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:53527 (local address 127.10.96.254:43373)
0927 10:45:28.897910 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.897914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.897927 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:28.898002 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.898005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.898048 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.898131 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.898503 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.898989 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.899751 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.899964 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.900070 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.902444 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:53527: BlockingRecv error: recv got EOF from 127.10.96.195:53527 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:45:28.902673 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.897729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:53527)
0927 10:45:28.897819 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.897834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.897916 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.898144 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.898147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.898301 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.898394 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.898400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.899088 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.899094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.899629 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.899636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.900071 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.900074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.900167 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.900585 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.900596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.902335 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:28 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:28.907683 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:28.903232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:47467)
0927 10:45:28.903334 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:28.903346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:28.903404 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:28.903559 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:28.903561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:28.903675 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:28.903740 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.903744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.904463 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.904465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.904971 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:28.904978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.905399 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.905401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:28.905506 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.905975 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:28.905986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:28.907597 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 10:45:28.907871 19507 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:28.903277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:47467 (local address 127.10.96.254:43373)
0927 10:45:28.903407 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:28.903411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:28.903424 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:28.903461 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:28.903463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:28.903494 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:28.903552 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:28.903858 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.904358 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.905077 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:28.905276 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:28.905378 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:28.907736 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:47467: BlockingRecv error: recv got EOF from 127.10.96.195:47467 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 10:45:28.907908 19770 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:45:29 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:29 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:29.550402 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:29.543506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:58779)
0927 10:45:29.543893 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:29.543910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:29.544025 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:29.544353 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:29.544355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:29.544592 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:29.544714 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:29.544721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.546012 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.546015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:29.546593 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:29.546602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.547328 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.547331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:29.547397 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:29.548017 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:29.548029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:29.550283 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":85,"threads_started":1}
W20260927 10:45:29.550690 19776 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:29.543593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:58779 (local address 127.10.96.254:43373)
0927 10:45:29.544019 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:29.544024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:29.544038 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:29.544137 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:29.544140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:29.544195 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:29.544331 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:29.544846 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.545879 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.546715 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.547172 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.547295 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:29.550488 (+ 3193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:58779: BlockingRecv error: recv got EOF from 127.10.96.193:58779 (error 108)
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":86,"threads_started":1}
Sep 27 10:45:29 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:29 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:29.730507 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:29.724850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:47119)
0927 10:45:29.725181 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:29.725193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:29.725334 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:29.725628 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:29.725631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:29.725836 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:29.725925 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:29.725930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.726693 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.726697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:29.727263 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:29.727270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.727722 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.727725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:29.727800 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:29.728290 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:29.728302 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:29.730390 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
W20260927 10:45:29.730662 19776 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:29.724964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:47119 (local address 127.10.96.254:43373)
0927 10:45:29.725135 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:29.725140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:29.725269 (+ 129us) server_negotiation.cc:407] Connection header received
0927 10:45:29.725434 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:29.725439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:29.725512 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:29.725603 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:29.726043 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.726578 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.727386 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.727612 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.727721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:29.730514 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:47119: BlockingRecv error: recv got EOF from 127.10.96.194:47119 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 10:45:29 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:29 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:29.914249 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:29.908531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:43563)
0927 10:45:29.908805 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:29.908821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:29.908935 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:29.909171 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:29.909173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:29.909353 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:29.909448 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:29.909454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.910224 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.910226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:29.910785 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:29.910820 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.911408 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.911412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:29.911482 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:29.912111 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:29.912123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:29.914092 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
W20260927 10:45:29.914402 19776 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:29.908600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:43563 (local address 127.10.96.254:43373)
0927 10:45:29.908749 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:29.908753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:29.908903 (+ 150us) server_negotiation.cc:407] Connection header received
0927 10:45:29.909015 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:29.909018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:29.909064 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:29.909139 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:29.909604 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.910124 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.910937 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:29.911241 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:29.911329 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:29.914255 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:43563: BlockingRecv error: recv got EOF from 127.10.96.195:43563 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 10:45:30 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:30 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:30.556895 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:30.551170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:47129)
0927 10:45:30.551474 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:30.551487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:30.551580 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:30.551856 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:30.551859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:30.552104 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:30.552257 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:30.552265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.553346 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.553349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:30.553938 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:30.553946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.554398 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.554400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:30.554469 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:30.554970 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:30.554982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:30.556775 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
W20260927 10:45:30.557075 19780 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:30.551294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:47129 (local address 127.10.96.254:43373)
0927 10:45:30.551552 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:30.551555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:30.551569 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:45:30.551666 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:30.551668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:30.551710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:30.551823 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:30.552387 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.553190 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.554039 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.554282 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.554364 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:30.556906 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:47129: BlockingRecv error: recv got EOF from 127.10.96.193:47129 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
Sep 27 10:45:30 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:30 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:30.736927 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:30.731345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:32803)
0927 10:45:30.731639 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:30.731653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:30.731753 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:30.731991 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:30.731993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:30.732164 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:30.732256 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:30.732262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.733053 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.733055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:30.733721 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:30.733730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.734184 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.734186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:30.734251 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:30.734730 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:30.734741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:30.736787 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
W20260927 10:45:30.737106 19780 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:30.731411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:32803 (local address 127.10.96.254:43373)
0927 10:45:30.731559 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:30.731564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:30.731727 (+ 163us) server_negotiation.cc:407] Connection header received
0927 10:45:30.731843 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:30.731846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:30.731888 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:30.731953 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:30.732403 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.732950 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.733858 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.734076 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.734167 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:30.736939 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:32803: BlockingRecv error: recv got EOF from 127.10.96.194:32803 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 10:45:30 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:30 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:30.920794 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:30.915003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:43101)
0927 10:45:30.915413 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:30.915427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:30.915524 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:30.915744 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:30.915746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:30.915918 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:30.916017 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:30.916022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.916834 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.916837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:30.917416 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:30.917425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.917979 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.917983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:30.918050 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:30.918590 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:30.918602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:30.920588 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
W20260927 10:45:30.920936 19780 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:30.915127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:43101 (local address 127.10.96.254:43373)
0927 10:45:30.915308 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:30.915314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:30.915510 (+ 196us) server_negotiation.cc:407] Connection header received
0927 10:45:30.915591 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:30.915594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:30.915642 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:30.915713 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:30.916146 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.916670 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.917619 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:30.917852 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:30.917949 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:30.920778 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:43101: BlockingRecv error: recv got EOF from 127.10.96.195:43101 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 10:45:31 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:31 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:31.563280 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:31.557653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:44955)
0927 10:45:31.557933 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:31.557955 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:31.558047 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:31.558281 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:31.558284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:31.558449 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:31.558540 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:31.558545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.559455 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.559461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:31.560174 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:31.560183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.560630 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.560634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:31.560717 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:31.561288 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:31.561300 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:31.563114 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":67,"threads_started":1}
W20260927 10:45:31.563366 19784 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:31.557791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:44955 (local address 127.10.96.254:43373)
0927 10:45:31.558017 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:31.558021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:31.558034 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:31.558128 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:31.558130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:31.558172 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:31.558247 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:31.558653 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.559347 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.560267 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.560501 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.560628 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:31.563230 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:44955: BlockingRecv error: recv got EOF from 127.10.96.193:44955 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
Sep 27 10:45:31 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:31 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:31.743009 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:31.737631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:36545)
0927 10:45:31.737901 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:31.737913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:31.738009 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:31.738221 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:31.738223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:31.738412 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:31.738504 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:31.738509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.739228 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.739231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:31.739794 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:31.739801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.740238 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.740240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:31.740302 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:31.740884 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:31.740895 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:31.742897 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
W20260927 10:45:31.743191 19784 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:31.737722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36545 (local address 127.10.96.254:43373)
0927 10:45:31.737858 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:31.737862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:31.737974 (+ 112us) server_negotiation.cc:407] Connection header received
0927 10:45:31.738073 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:31.738075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:31.738118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:31.738189 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:31.738602 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.739132 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.739915 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.740135 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.740225 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:31.743051 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:36545: BlockingRecv error: recv got EOF from 127.10.96.194:36545 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:45:31 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:31 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:31.926961 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:31.921706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:47539)
0927 10:45:31.921995 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:31.922008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:31.922109 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:31.922364 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:31.922367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:31.922540 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:31.922636 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:31.922640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.923448 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.923451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:31.924004 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:31.924011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.924439 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.924441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:31.924500 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:31.924970 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:31.924981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:31.926848 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
W20260927 10:45:31.927088 19784 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:31.921771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:47539 (local address 127.10.96.254:43373)
0927 10:45:31.921918 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:31.921924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:31.922066 (+ 142us) server_negotiation.cc:407] Connection header received
0927 10:45:31.922175 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:31.922177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:31.922223 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:31.922297 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:31.922798 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.923327 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.924114 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:31.924339 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:31.924427 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:31.926969 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:47539: BlockingRecv error: recv got EOF from 127.10.96.195:47539 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 10:45:32 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:32 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:32.569675 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:32.564160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:53323)
0927 10:45:32.564463 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:32.564476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:32.564578 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:32.564841 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:32.564843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:32.565022 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:32.565119 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:32.565124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.565870 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.565872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:32.566435 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:32.566443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.566838 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.566840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:32.566898 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:32.567484 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:32.567522 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:32.569504 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
W20260927 10:45:32.569738 19788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:32.564275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:53323 (local address 127.10.96.254:43373)
0927 10:45:32.564540 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:32.564544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:32.564559 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:45:32.564668 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:32.564670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:32.564713 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:32.564793 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:32.565220 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.565774 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.566533 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.566744 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.566818 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:32.569621 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:53323: BlockingRecv error: recv got EOF from 127.10.96.193:53323 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
Sep 27 10:45:32 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:32 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:32.756474 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:32.743790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:55925)
0927 10:45:32.744096 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:32.744110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:32.744222 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:32.744461 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:32.744463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:32.744635 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:32.744726 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:32.744731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.745596 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.745599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:32.746155 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:32.746163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.746603 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.746605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:32.746671 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:32.753981 (+ 7310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:32.754004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:32.756265 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
W20260927 10:45:32.756628 19788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:32.743845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:55925 (local address 127.10.96.254:43373)
0927 10:45:32.743992 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:32.743997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:32.744195 (+ 198us) server_negotiation.cc:407] Connection header received
0927 10:45:32.744309 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:32.744311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:32.744355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:32.744421 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:32.744867 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.745409 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.746278 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.746500 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.746581 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:32.756487 (+ 9906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:55925: BlockingRecv error: recv got EOF from 127.10.96.194:55925 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 10:45:32 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:32 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:32.933486 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:32.927890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:34973)
0927 10:45:32.928195 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:32.928207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:32.928313 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:32.928575 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:32.928577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:32.928777 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:32.928898 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:32.928906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.929833 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.929837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:32.930599 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:32.930613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.931030 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.931033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:32.931096 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:32.931617 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:32.931629 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:32.933349 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
W20260927 10:45:32.933667 19788 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:32.927973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:34973 (local address 127.10.96.254:43373)
0927 10:45:32.928118 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:32.928122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:32.928283 (+ 161us) server_negotiation.cc:407] Connection header received
0927 10:45:32.928405 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:32.928408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:32.928453 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:32.928531 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:32.929016 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.929710 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.930707 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:32.930928 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:32.931007 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:32.933540 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:34973: BlockingRecv error: recv got EOF from 127.10.96.195:34973 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 10:45:33 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:33 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:33.576365 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:33.570551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:40041)
0927 10:45:33.570843 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:33.570860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:33.570974 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:33.571567 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:33.571570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:33.571837 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:33.571954 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:33.571962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.572734 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.572737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:33.573317 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:33.573325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.573753 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.573756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:33.573821 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:33.574396 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:33.574407 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:33.576228 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
W20260927 10:45:33.576563 19792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:33.570600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:40041 (local address 127.10.96.254:43373)
0927 10:45:33.570922 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:33.570926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:33.570943 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:45:33.571196 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:33.571200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:33.571256 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:33.571341 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:33.572072 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.572628 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.573425 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.573659 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.573737 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:33.576383 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:40041: BlockingRecv error: recv got EOF from 127.10.96.193:40041 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":92,"threads_started":1}
Sep 27 10:45:33 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:33 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:33.762897 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:33.757298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:42625)
0927 10:45:33.757583 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:33.757596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:33.757677 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:33.757903 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:33.757905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:33.758078 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:33.758167 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:33.758172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.758940 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.758945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:33.759575 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:33.759584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.760020 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.760023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:33.760089 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:33.760647 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:33.760659 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:33.762777 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
W20260927 10:45:33.763092 19792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:33.757543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:42625 (local address 127.10.96.254:43373)
0927 10:45:33.757669 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:33.757678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:33.757691 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:33.757746 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:33.757748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:33.757788 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:33.757868 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:33.758266 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.758798 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.759689 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.759911 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.759994 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:33.762930 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:42625: BlockingRecv error: recv got EOF from 127.10.96.194:42625 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:45:33 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:33 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 10:45:33.939844 19792 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:33.934421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56913 (local address 127.10.96.254:43373)
0927 10:45:33.934574 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:33.934579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:33.934696 (+ 117us) server_negotiation.cc:407] Connection header received
0927 10:45:33.934818 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:33.934821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:33.934878 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:33.934966 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:33.935413 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.935931 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.936738 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.936965 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.937074 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:33.939746 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:56913: BlockingRecv error: recv got EOF from 127.10.96.195:56913 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:45:33.939836 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:33.934261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:56913)
0927 10:45:33.934601 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:33.934616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:33.934731 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:33.934989 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:33.934992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:33.935205 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:33.935292 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:33.935298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.936051 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.936054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:33.936622 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:33.936629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:33.937075 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:33.937079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:33.937157 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:33.937774 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:33.937788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:33.939627 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":100,"threads_started":1}
Sep 27 10:45:34 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:34 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:34.582631 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:34.577071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:40875)
0927 10:45:34.577382 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:34.577395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:34.577539 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:34.577802 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:34.577804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:34.577981 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:34.578072 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:34.578077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.578913 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.578916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:34.579506 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:34.579515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.579946 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.579948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:34.580011 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:34.580536 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:34.580547 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:34.582465 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
W20260927 10:45:34.582693 19796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:34.577144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:40875 (local address 127.10.96.254:43373)
0927 10:45:34.577447 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:34.577451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:34.577481 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:45:34.577634 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:34.577636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:34.577681 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:34.577759 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:34.578233 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.578784 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.579615 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.579848 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.579926 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:34.582578 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:40875: BlockingRecv error: recv got EOF from 127.10.96.193:40875 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
Sep 27 10:45:34 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:34 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:34.769390 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:34.763669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:50105)
0927 10:45:34.763960 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:34.763974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:34.764080 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:34.764335 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:34.764338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:34.764508 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:34.764603 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:34.764608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.765331 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.765333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:34.765900 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:34.765907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.766635 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.766638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:34.766731 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:34.767329 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:34.767344 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:34.769230 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
W20260927 10:45:34.769511 19796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:34.763753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:50105 (local address 127.10.96.254:43373)
0927 10:45:34.763937 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:34.763941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:34.764030 (+ 89us) server_negotiation.cc:407] Connection header received
0927 10:45:34.764146 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:34.764148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:34.764191 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:34.764258 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:34.764705 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.765215 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.766028 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.766373 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.766511 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:34.769373 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:50105: BlockingRecv error: recv got EOF from 127.10.96.194:50105 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 10:45:34 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:34 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:34.946352 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:34.940666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:42961)
0927 10:45:34.940979 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:34.940997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:34.941113 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:34.941410 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:34.941413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:34.941652 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:34.941746 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:34.941752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.942553 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.942558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:34.943124 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:34.943132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.943593 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.943596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:34.943694 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:34.944296 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:34.944309 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:34.946179 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
W20260927 10:45:34.946447 19796 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:34.940802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:42961 (local address 127.10.96.254:43373)
0927 10:45:34.940967 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:34.940971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:34.941059 (+ 88us) server_negotiation.cc:407] Connection header received
0927 10:45:34.941227 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:34.941231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:34.941295 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:34.941386 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:34.941880 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.942440 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.943244 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:34.943479 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:34.943594 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:34.946326 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:42961: BlockingRecv error: recv got EOF from 127.10.96.195:42961 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 10:45:35 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:35 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:35.589036 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:35.583475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:33143)
0927 10:45:35.583839 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:35.583857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:35.583976 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:35.584243 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:35.584246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:35.584424 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:35.584516 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:35.584521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.585307 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.585311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:35.585916 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:35.585924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.586380 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.586383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:35.586446 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:35.586942 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:35.586954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:35.588871 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":79,"threads_started":1}
W20260927 10:45:35.589110 19800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:35.583607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:33143 (local address 127.10.96.254:43373)
0927 10:45:35.583850 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:35.583854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:35.583967 (+ 113us) server_negotiation.cc:407] Connection header received
0927 10:45:35.584070 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:35.584072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:35.584116 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:35.584192 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:35.584644 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.585195 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.586028 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.586257 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.586363 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:35.588986 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:33143: BlockingRecv error: recv got EOF from 127.10.96.193:33143 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":55,"threads_started":1}
Sep 27 10:45:35 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:35 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:35.775665 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:35.770174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:45963)
0927 10:45:35.770519 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:35.770532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:35.770660 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:35.770877 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:35.770879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:35.771067 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:35.771198 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:35.771203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.771972 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.771974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:35.772534 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:35.772541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.772949 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.772951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:35.773011 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:35.773532 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:35.773547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:35.775466 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
W20260927 10:45:35.775693 19800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:35.770270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:45963 (local address 127.10.96.254:43373)
0927 10:45:35.770428 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:35.770433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:35.770646 (+ 213us) server_negotiation.cc:407] Connection header received
0927 10:45:35.770741 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:35.770744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:35.770790 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:35.770869 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:35.771295 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.771879 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.772634 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.772857 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.772934 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:35.775604 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:45963: BlockingRecv error: recv got EOF from 127.10.96.194:45963 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 10:45:35 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:35 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:35.952603 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:35.947111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:57913)
0927 10:45:35.947453 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:35.947465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:35.947563 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:35.947805 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:35.947807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:35.947989 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:35.948081 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:35.948086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.948987 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.948989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:35.949565 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:35.949573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.949991 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.949994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:35.950056 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:35.950555 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:35.950567 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:35.952407 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
W20260927 10:45:35.952623 19800 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:35.947206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:57913 (local address 127.10.96.254:43373)
0927 10:45:35.947377 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:35.947382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:35.947546 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:45:35.947643 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:35.947646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:35.947691 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:35.947763 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:35.948192 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.948884 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.949667 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:35.949888 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:35.949966 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:35.952525 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:57913: BlockingRecv error: recv got EOF from 127.10.96.195:57913 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 10:45:36 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:36 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:36.595670 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:36.589822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:39403)
0927 10:45:36.590098 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:36.590111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:36.590200 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:36.590464 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:36.590466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:36.590641 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:36.590733 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:36.590738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.591730 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.591733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:36.592580 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:36.592590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.593041 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.593044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:36.593104 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:36.593655 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:36.593667 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:36.595557 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
W20260927 10:45:36.595875 19804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:36.589975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:39403 (local address 127.10.96.254:43373)
0927 10:45:36.590187 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:36.590190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:36.590202 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:36.590304 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:36.590306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:36.590349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:36.590427 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:36.590843 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.591638 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.592702 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.592934 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.593022 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:36.595715 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:39403: BlockingRecv error: recv got EOF from 127.10.96.193:39403 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
Sep 27 10:45:36 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:36 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:36.782554 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:36.776514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:40833)
0927 10:45:36.776829 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:36.776841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:36.776947 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:36.777252 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:36.777255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:36.777435 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:36.777595 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:36.777602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.778849 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.778852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:36.779409 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:36.779417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.779860 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.779862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:36.779922 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:36.780419 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:36.780430 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:36.782437 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
W20260927 10:45:36.782694 19804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:36.776603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:40833 (local address 127.10.96.254:43373)
0927 10:45:36.776777 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:36.776782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:36.776946 (+ 164us) server_negotiation.cc:407] Connection header received
0927 10:45:36.777018 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:36.777021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:36.777068 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:36.777149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:36.777694 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.778732 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.779531 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.779747 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.779839 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:36.782575 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:40833: BlockingRecv error: recv got EOF from 127.10.96.194:40833 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 10:45:36 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:36 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:36.958647 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:36.953415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:39717)
0927 10:45:36.953729 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:36.953742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:36.953849 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:36.954072 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:36.954074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:36.954242 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:36.954343 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:36.954348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.955064 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.955068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:36.955627 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:36.955635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.956088 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.956092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:36.956176 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:36.956671 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:36.956682 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:36.958534 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":155,"threads_started":1}
W20260927 10:45:36.958817 19804 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:36.953518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:39717 (local address 127.10.96.254:43373)
0927 10:45:36.953658 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:36.953663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:36.953819 (+ 156us) server_negotiation.cc:407] Connection header received
0927 10:45:36.953927 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:36.953929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:36.953975 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:36.954040 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:36.954436 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.954945 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.955743 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:36.955974 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:36.956083 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:36.958662 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:39717: BlockingRecv error: recv got EOF from 127.10.96.195:39717 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 10:45:37 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:37 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:37.602267 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:37.596489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:34925)
0927 10:45:37.596822 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:37.596841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:37.596949 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:37.597207 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:37.597209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:37.597440 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:37.597629 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:37.597636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.598412 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.598415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:37.599036 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:37.599044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.599491 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.599494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:37.599562 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:37.600096 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:37.600107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:37.602144 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":88,"threads_started":1}
W20260927 10:45:37.602468 19808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:37.596607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:34925 (local address 127.10.96.254:43373)
0927 10:45:37.596857 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:37.596860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:37.596897 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:45:37.597038 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:37.597041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:37.597082 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:37.597161 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:37.597733 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.598286 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.599138 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.599364 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.599445 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:37.602310 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:34925: BlockingRecv error: recv got EOF from 127.10.96.193:34925 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":57,"threads_started":1}
Sep 27 10:45:37 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:37 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:37.792801 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:37.783357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:47031)
0927 10:45:37.783672 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:37.783689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:37.783809 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:37.784100 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:37.784103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:37.784345 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:37.784436 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:37.784442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.785212 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.785217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:37.785808 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:37.785816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.786259 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.786263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:37.786354 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:37.790412 (+ 4058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:37.790465 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:37.792676 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
W20260927 10:45:37.793011 19808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:37.783493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:47031 (local address 127.10.96.254:43373)
0927 10:45:37.783665 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:37.783670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:37.783750 (+ 80us) server_negotiation.cc:407] Connection header received
0927 10:45:37.783931 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:37.783934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:37.783989 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:37.784075 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:37.784555 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.785103 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.785926 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.786150 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.786260 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:37.792884 (+ 6624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:47031: BlockingRecv error: recv got EOF from 127.10.96.194:47031 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 10:45:37 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:37 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:37.965258 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:37.959424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:33851)
0927 10:45:37.959758 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:37.959778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:37.959927 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:37.960235 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:37.960238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:37.960442 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:37.960533 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:37.960539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.961301 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.961303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:37.961917 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:37.961927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.962394 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.962397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:37.962467 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:37.962984 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:37.962996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:37.965074 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
W20260927 10:45:37.965317 19808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:37.959565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:33851 (local address 127.10.96.254:43373)
0927 10:45:37.959737 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:37.959742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:37.959863 (+ 121us) server_negotiation.cc:407] Connection header received
0927 10:45:37.960029 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:37.960034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:37.960135 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:37.960214 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:37.960655 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.961181 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.962027 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:37.962240 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:37.962323 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:37.965193 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:33851: BlockingRecv error: recv got EOF from 127.10.96.195:33851 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 10:45:38 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:38 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:38.609249 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:38.603245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:55971)
0927 10:45:38.603513 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:38.603527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:38.603609 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:38.603902 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:38.603904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:38.604077 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:38.604167 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:38.604173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.604975 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.604978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:38.605775 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:38.605803 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.606431 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.606434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:38.606494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:38.607237 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:38.607248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:38.609129 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260927 10:45:38.609355 19812 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:38.603317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55971 (local address 127.10.96.254:43373)
0927 10:45:38.603559 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:38.603562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:38.603575 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:38.603729 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:38.603732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:38.603781 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:38.603870 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:38.604278 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.604860 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.605929 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.606317 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.606421 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:38.609250 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:55971: BlockingRecv error: recv got EOF from 127.10.96.193:55971 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
Sep 27 10:45:38 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:38 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:38.799129 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:38.793679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:47887)
0927 10:45:38.794024 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:38.794036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:38.794157 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:38.794421 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:38.794424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:38.794628 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:38.794712 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:38.794718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.795453 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.795460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:38.796017 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:38.796024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.796462 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.796465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:38.796543 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:38.797071 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:38.797082 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:38.799013 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":91,"threads_started":1}
W20260927 10:45:38.799278 19812 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:38.793739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:47887 (local address 127.10.96.254:43373)
0927 10:45:38.793918 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:38.793923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:38.794114 (+ 191us) server_negotiation.cc:407] Connection header received
0927 10:45:38.794261 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:38.794264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:38.794315 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:38.794400 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:38.794827 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.795348 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.796130 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.796354 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.796462 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:38.799154 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:47887: BlockingRecv error: recv got EOF from 127.10.96.194:47887 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 10:45:38 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:38 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:38.971500 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:38.966081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:53325)
0927 10:45:38.966372 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:38.966385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:38.966476 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:38.966699 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:38.966701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:38.966880 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:38.966977 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:38.966983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.967713 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.967716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:38.968293 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:38.968301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.968694 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.968696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:38.968757 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:38.969320 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:38.969332 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:38.971389 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
W20260927 10:45:38.971700 19812 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:38.966217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:53325 (local address 127.10.96.254:43373)
0927 10:45:38.966348 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:38.966352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:38.966447 (+ 95us) server_negotiation.cc:407] Connection header received
0927 10:45:38.966551 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:38.966553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:38.966593 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:38.966660 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:38.967074 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.967617 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.968395 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:38.968607 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:38.968682 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:38.971513 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:53325: BlockingRecv error: recv got EOF from 127.10.96.195:53325 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:45:39 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:39 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:39.615731 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:39.610085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:33365)
0927 10:45:39.610347 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:39.610360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:39.610447 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:39.610710 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:39.610712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:39.610878 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:39.610977 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:39.610982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.611876 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.611879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:39.612529 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:39.612539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.612987 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.612989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:39.613048 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:39.613558 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:39.613568 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:39.615571 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260927 10:45:39.615864 19816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:39.610164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:33365 (local address 127.10.96.254:43373)
0927 10:45:39.610468 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:39.610474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:39.610494 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:45:39.610534 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:39.610537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:39.610585 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:39.610677 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:39.611097 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.611766 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.612659 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.612889 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.612974 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:39.615713 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:33365: BlockingRecv error: recv got EOF from 127.10.96.193:33365 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":117,"threads_started":1}
Sep 27 10:45:39 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:39 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:39.805655 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:39.799993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:49865)
0927 10:45:39.800344 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:39.800356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:39.800491 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:39.800774 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:39.800777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:39.800985 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:39.801074 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:39.801079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.801890 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.801895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:39.802486 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:39.802495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.802944 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.802947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:39.803038 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:39.803607 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:39.803618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:39.805505 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
W20260927 10:45:39.805724 19816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:39.800011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:49865 (local address 127.10.96.254:43373)
0927 10:45:39.800191 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:39.800195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:39.800442 (+ 247us) server_negotiation.cc:407] Connection header received
0927 10:45:39.800607 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:39.800611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:39.800661 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:39.800747 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:39.801190 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.801784 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.802612 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.802832 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.802941 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:39.805634 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:49865: BlockingRecv error: recv got EOF from 127.10.96.194:49865 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 10:45:39 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:39 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:39.978180 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:39.972287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:37791)
0927 10:45:39.972553 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:39.972574 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:39.972669 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:39.972905 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:39.972907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:39.973077 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:39.973173 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:39.973179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.974240 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.974244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:39.974951 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:39.974961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.975402 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.975404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:39.975467 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:39.976079 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:39.976106 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:39.978056 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":105,"threads_started":1}
W20260927 10:45:39.978323 19816 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:39.972328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:37791 (local address 127.10.96.254:43373)
0927 10:45:39.972459 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:39.972469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:39.972640 (+ 171us) server_negotiation.cc:407] Connection header received
0927 10:45:39.972744 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:39.972746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:39.972794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:39.972869 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:39.973327 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.974107 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.975073 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:39.975290 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:39.975387 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:39.978175 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:37791: BlockingRecv error: recv got EOF from 127.10.96.195:37791 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:45:40 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:40 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:40.621836 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:40.616569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:54653)
0927 10:45:40.616877 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:40.616893 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:40.617015 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:40.617294 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:40.617297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:40.617499 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:40.617590 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:40.617595 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.618295 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.618299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:40.618860 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:40.618869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.619260 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.619262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:40.619328 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:40.619830 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:40.619840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:40.621672 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
W20260927 10:45:40.621913 19820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:40.616628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:54653 (local address 127.10.96.254:43373)
0927 10:45:40.616892 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:40.616896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:40.616978 (+ 82us) server_negotiation.cc:407] Connection header received
0927 10:45:40.617120 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:40.617122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:40.617166 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:40.617240 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:40.617685 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.618195 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.618964 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.619173 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.619244 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:40.621794 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:54653: BlockingRecv error: recv got EOF from 127.10.96.193:54653 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
Sep 27 10:45:40 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:40 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:40.813639 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:40.806532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:54495)
0927 10:45:40.806794 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:40.806810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:40.806922 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:40.807238 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:40.807244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:40.807637 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:40.807768 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:40.807777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.808534 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.808537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:40.809309 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:40.809318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.809824 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.809828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:40.809904 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:40.810695 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:40.810712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:40.813397 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
W20260927 10:45:40.813746 19820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:40.806879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:54495 (local address 127.10.96.254:43373)
0927 10:45:40.807002 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:40.807006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:40.807018 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:45:40.807064 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:40.807067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:40.807116 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:40.807202 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:40.807880 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.808416 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.809442 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.809714 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.809832 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:40.813613 (+ 3781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:54495: BlockingRecv error: recv got EOF from 127.10.96.194:54495 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 10:45:40 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:40 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:40.985173 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:40.978868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:41493)
0927 10:45:40.979183 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:40.979204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:40.979354 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:40.979622 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:40.979625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:40.979839 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:40.979941 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:40.979947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.980699 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.980704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:40.981519 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:40.981529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.982031 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.982035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:40.982118 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:40.982661 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:40.982673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:40.985002 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
W20260927 10:45:40.985231 19820 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:40.978984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:41493 (local address 127.10.96.254:43373)
0927 10:45:40.979139 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:40.979144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:40.979288 (+ 144us) server_negotiation.cc:407] Connection header received
0927 10:45:40.979456 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:40.979461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:40.979513 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:40.979599 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:40.980066 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.980592 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.981669 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:40.981917 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:40.982030 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:40.985126 (+ 3096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:41493: BlockingRecv error: recv got EOF from 127.10.96.195:41493 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 10:45:41 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:41 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:41.628649 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:41.622713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:58215)
0927 10:45:41.623017 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:41.623034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:41.623153 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:41.623387 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:41.623389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:41.623563 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:41.623653 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:41.623657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.624445 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.624447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:41.625341 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:41.625351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.625924 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.625927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:41.625988 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:41.626628 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:41.626640 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:41.628481 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
W20260927 10:45:41.628719 19824 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:41.622893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:58215 (local address 127.10.96.254:43373)
0927 10:45:41.623159 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:41.623162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:41.623177 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:45:41.623226 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:41.623228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:41.623269 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:41.623376 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:41.623758 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.624308 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.625484 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.625810 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.625905 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:41.628598 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:58215: BlockingRecv error: recv got EOF from 127.10.96.193:58215 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
Sep 27 10:45:41 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:41 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:41.820174 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:41.814576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:40411)
0927 10:45:41.814861 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:41.814874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:41.814961 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:41.815247 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:41.815249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:41.815439 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:41.815534 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:41.815540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.816331 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.816333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:41.816921 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:41.816929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.817372 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.817375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:41.817434 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:41.817952 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:41.817963 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:41.819925 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
W20260927 10:45:41.820212 19824 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:41.814780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:40411 (local address 127.10.96.254:43373)
0927 10:45:41.814918 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:41.814922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:41.814935 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:41.815051 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:41.815053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:41.815094 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:41.815161 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:41.815663 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.816225 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.817027 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.817238 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.817339 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:41.820096 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:40411: BlockingRecv error: recv got EOF from 127.10.96.194:40411 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 10:45:41 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:41 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:41.991362 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:41.986032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:48955)
0927 10:45:41.986305 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:41.986318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:41.986416 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:41.986683 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:41.986685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:41.986871 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:41.986964 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:41.986970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.987680 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.987682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:41.988262 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:41.988276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.988692 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.988694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:41.988753 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:41.989255 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:41.989272 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:41.991213 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
W20260927 10:45:41.991482 19824 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:41.986116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:48955 (local address 127.10.96.254:43373)
0927 10:45:41.986247 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:41.986252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:41.986388 (+ 136us) server_negotiation.cc:407] Connection header received
0927 10:45:41.986493 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:41.986496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:41.986541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:41.986614 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:41.987066 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.987586 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.988379 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:41.988595 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:41.988676 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:41.991339 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:48955: BlockingRecv error: recv got EOF from 127.10.96.195:48955 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 10:45:42 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:42 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:42.635601 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:42.629695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:47081)
0927 10:45:42.630033 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:42.630046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:42.630127 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:42.630582 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:42.630586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:42.630855 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:42.630976 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:42.630985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.631928 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.631931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:42.632489 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:42.632497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.632916 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.632918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:42.632977 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:42.633529 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:42.633545 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:42.635480 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
W20260927 10:45:42.635787 19828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:42.629909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:47081 (local address 127.10.96.254:43373)
0927 10:45:42.630340 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:42.630344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:42.630357 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:45:42.630399 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:42.630402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:42.630446 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:42.630533 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:42.631108 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.631824 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.632592 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.632819 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.632896 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:42.635643 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:47081: BlockingRecv error: recv got EOF from 127.10.96.193:47081 (error 108)
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":68,"threads_started":1}
Sep 27 10:45:42 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:42 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:42.829806 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:42.820988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:36125)
0927 10:45:42.821356 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:42.821369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:42.821530 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:42.821838 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:42.821841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:42.822051 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:42.822138 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:42.822143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.822957 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.822961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:42.823547 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:42.823555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.824059 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.824063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:42.824148 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:42.827779 (+ 3631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:42.827792 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:42.829679 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
W20260927 10:45:42.829994 19828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:42.821108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36125 (local address 127.10.96.254:43373)
0927 10:45:42.821290 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:42.821296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:42.821448 (+ 152us) server_negotiation.cc:407] Connection header received
0927 10:45:42.821633 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:42.821637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:42.821700 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:42.821817 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:42.822254 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.822853 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.823668 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.823949 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.824058 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:42.829820 (+ 5762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:36125: BlockingRecv error: recv got EOF from 127.10.96.194:36125 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 10:45:42 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:42 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.195@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:42.997989 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:42.992129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.195:51099)
0927 10:45:42.992406 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:42.992420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:42.992515 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:42.992747 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:42.992749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:42.992936 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:42.993029 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:42.993035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.993841 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.993845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:42.994497 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:42.994507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.994917 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.994920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:42.994988 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:42.995639 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:42.995658 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:42.997860 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":117,"threads_started":1}
W20260927 10:45:42.998106 19828 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:42.992200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:51099 (local address 127.10.96.254:43373)
0927 10:45:42.992352 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:42.992364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:42.992494 (+ 130us) server_negotiation.cc:407] Connection header received
0927 10:45:42.992594 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:42.992597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:42.992643 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:42.992713 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:42.993133 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.993712 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.994611 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:42.994817 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:42.994894 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:42.997971 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.195:51099: BlockingRecv error: recv got EOF from 127.10.96.195:51099 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 10:45:43 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:43 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.193@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:43.642058 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:43.636515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.193:34693)
0927 10:45:43.636774 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:43.636787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:43.636906 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:43.637206 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:43.637209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:43.637411 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:43.637570 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:43.637577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.638299 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.638301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:43.638863 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:43.638870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.639332 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.639335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:43.639403 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:43.640002 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:43.640026 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:43.641944 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
W20260927 10:45:43.642211 19832 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:43.636541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:34693 (local address 127.10.96.254:43373)
0927 10:45:43.636800 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:43.636803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:43.636856 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:45:43.637037 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:43.637040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:43.637087 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:43.637185 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:43.637682 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.638211 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.638969 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.639190 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.639269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:43.642080 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.193:34693: BlockingRecv error: recv got EOF from 127.10.96.193:34693 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":74,"threads_started":1}
Sep 27 10:45:43 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 10:45:43 dist-test-slave-wmg0 krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.96.194@KRBTEST.COM for oryx/127.10.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 10:45:43.835644 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:43.830598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43373 (local address 127.10.96.194:38177)
0927 10:45:43.830882 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:43.830895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:43.830985 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:43.831213 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:43.831215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:43.831390 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:43.831477 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:43.831482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.832195 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.832197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:43.832729 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:43.832736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.833162 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.833165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:43.833222 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:43.833695 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:43.833706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:43.835522 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.96.254:43373: Server oryx/127.10.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":114,"threads_started":1}
W20260927 10:45:43.835796 19832 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:43.830687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:38177 (local address 127.10.96.254:43373)
0927 10:45:43.830826 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:43.830830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:43.830965 (+ 135us) server_negotiation.cc:407] Connection header received
0927 10:45:43.831065 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:43.831067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:43.831108 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:43.831179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:43.831576 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.832106 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.832833 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:43.833068 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:43.833145 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:43.835649 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.96.194:38177: BlockingRecv error: recv got EOF from 127.10.96.194:38177 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:45:43.897454 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19357
I20260927 10:45:43.903226 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19509
I20260927 10:45:43.908558 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19641
I20260927 10:45:43.914454 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19287
2026-09-27T10:45:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17890 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T10:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:43Z Disabled control of system clock
I20260927 10:45:43.943882 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43351
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 10:45:44.056367 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:44.056628 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:44.056663 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:44.060685 19837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:44.060738 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:44.060753 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:44.060770 19837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:44.060784 19837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:44.065477 19837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:44.066552 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:44.067921 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:44.076462 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.076620 19837 server_base.cc:1034] running on GCE node
W20260927 10:45:44.076474 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.076552 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.077265 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:44.078365 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:44.079615 19837 hybrid_clock.cc:648] HybridClock initialized: now 1790505944079577 us; error 43 us; skew 500 ppm
I20260927 10:45:44.081517 19837 webserver.cc:460] Webserver started at http://127.10.96.254:40221/ using document root <none> and password file <none>
I20260927 10:45:44.082103 19837 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:44.082165 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:44.082392 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:44.084435 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "574977cf4ba845d39a3dcf2ec4a6e462"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "78eae644fd254de544728d0a21eddb17"
server_key_iv: "e05d49f8de0ff63afd38465ca7f11818"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.085000 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "574977cf4ba845d39a3dcf2ec4a6e462"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "78eae644fd254de544728d0a21eddb17"
server_key_iv: "e05d49f8de0ff63afd38465ca7f11818"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.088542 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:45:44.090857 19851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.092108 19837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:44.092242 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "574977cf4ba845d39a3dcf2ec4a6e462"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "78eae644fd254de544728d0a21eddb17"
server_key_iv: "e05d49f8de0ff63afd38465ca7f11818"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.092392 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:44.115448 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:44.116163 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:44.116375 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:44.123273 19837 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43351
I20260927 10:45:44.123337 19903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43351 every 8 connection(s)
I20260927 10:45:44.124269 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:44.127400 19904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:44.129388 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19837
I20260927 10:45:44.129515 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:44.129796 10627 external_mini_cluster.cc:1428] Setting key 52c0cc6ed70f67cf6e58a7200bc7f13d
I20260927 10:45:44.133512 19904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462: Bootstrap starting.
I20260927 10:45:44.135645 19904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:44.136399 19904 log.cc:824] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:44.138425 19904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462: No bootstrap required, opened a new log
I20260927 10:45:44.139127 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.131259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.10.96.254:43351)
0927 10:45:44.131895 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:44.131910 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:44.131943 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:45:44.132673 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:44.132690 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:44.133078 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:44.133434 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:44.133872 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.134796 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.135669 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.136201 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.136383 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.137239 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:44.137245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:44.137304 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:44.137747 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:44.137807 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:44.137828 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:44.137936 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:44.138093 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 10:45:44.138348 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260927 10:45:44.142277 19904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43351 } }
I20260927 10:45:44.142522 19904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:44.142608 19904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574977cf4ba845d39a3dcf2ec4a6e462, State: Initialized, Role: FOLLOWER
I20260927 10:45:44.143010 19904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43351 } }
I20260927 10:45:44.143157 19904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:44.143239 19904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:44.143359 19904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:44.144236 19904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43351 } }
I20260927 10:45:44.144565 19904 leader_election.cc:302] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 574977cf4ba845d39a3dcf2ec4a6e462; no voters:
I20260927 10:45:44.144850 19904 leader_election.cc:288] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:44.144959 19909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:44.145195 19909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [term 1 LEADER]: Becoming Leader. State: Replica: 574977cf4ba845d39a3dcf2ec4a6e462, State: Running, Role: LEADER
I20260927 10:45:44.145604 19909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43351 } }
I20260927 10:45:44.146126 19904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:44.147228 19910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43351 } } }
I20260927 10:45:44.147372 19910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:44.147635 19911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574977cf4ba845d39a3dcf2ec4a6e462. Latest consensus state: current_term: 1 leader_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574977cf4ba845d39a3dcf2ec4a6e462" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43351 } } }
I20260927 10:45:44.147773 19918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:44.147778 19911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:44.150450 19918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:44.155613 19918 catalog_manager.cc:1347] Generated new cluster ID: f86bfb889dc34980a2550a36c79241d1
I20260927 10:45:44.155676 19918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:44.163566 19918 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:44.164697 19918 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:44.171350 19918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 574977cf4ba845d39a3dcf2ec4a6e462: Generated new TSK 0
I20260927 10:45:44.172045 19918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:44.176707 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43351
--builtin_ntp_servers=127.10.96.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 10:45:44.297684 19928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:44.297983 19928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:44.298061 19928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:44.302120 19928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:44.302213 19928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:44.302304 19928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:44.307063 19928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:44.308188 19928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:44.309617 19928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:44.319051 19936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.319063 19934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.319052 19933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.319355 19928 server_base.cc:1034] running on GCE node
I20260927 10:45:44.319820 19928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:44.320385 19928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:44.321552 19928 hybrid_clock.cc:648] HybridClock initialized: now 1790505944321531 us; error 25 us; skew 500 ppm
I20260927 10:45:44.323467 19928 webserver.cc:460] Webserver started at http://127.10.96.193:46431/ using document root <none> and password file <none>
I20260927 10:45:44.324036 19928 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:44.324097 19928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:44.324342 19928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:44.326449 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "ddf93355de9541fbbdbd31fcea826a39"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "3184c18d1e1c97615599809ae1972fce"
server_key_iv: "b881600ab5007b7cf81e0700d900d007"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.326998 19928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "ddf93355de9541fbbdbd31fcea826a39"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "3184c18d1e1c97615599809ae1972fce"
server_key_iv: "b881600ab5007b7cf81e0700d900d007"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.330551 19928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 10:45:44.332896 19942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.334157 19928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:44.334290 19928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "ddf93355de9541fbbdbd31fcea826a39"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "3184c18d1e1c97615599809ae1972fce"
server_key_iv: "b881600ab5007b7cf81e0700d900d007"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.334419 19928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:44.354525 19928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:44.355224 19928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:44.355429 19928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:44.356005 19928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:44.356930 19928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:44.357007 19928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.357096 19928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:44.357146 19928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.365833 19928 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:43133
I20260927 10:45:44.365865 20055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:43133 every 8 connection(s)
I20260927 10:45:44.366724 19928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:44.374096 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 19928
I20260927 10:45:44.374212 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:44.374511 10627 external_mini_cluster.cc:1428] Setting key 1baeeba73436bd4b7fb3aab0cbbd05e4
I20260927 10:45:44.375747 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.368602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:39391 (local address 127.10.96.254:43351)
0927 10:45:44.368736 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:44.368740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:44.369767 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 10:45:44.371103 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:44.371105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:44.371152 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:44.371227 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:44.372050 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.372586 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.373732 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.373966 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.374048 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.375231 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:44.375236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:44.375239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:44.375253 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:44.375273 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:44.375283 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:44.375386 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:44.375548 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:45:44.375595 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:45:44.376545 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43351
--builtin_ntp_servers=127.10.96.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:45:44.376771 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.368955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43351 (local address 127.10.96.193:39391)
0927 10:45:44.369645 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:44.369684 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:44.370908 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:44.371384 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:44.371391 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:44.371468 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:44.371933 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:44.371946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.372692 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.372694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:44.373632 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:44.373641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.374129 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.374134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:44.374342 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.374949 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:44.374950 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:44.374987 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:44.375068 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:44.375403 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:44.375412 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:44.375418 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:45:44.375542 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 10:45:44.375810 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260927 10:45:44.377991 20056 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43351
I20260927 10:45:44.378222 20056 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:44.378726 20056 heartbeater.cc:511] Master 127.10.96.254:43351 requested a full tablet report, sending...
I20260927 10:45:44.380366 19868 ts_manager.cc:194] Registered new tserver with Master: ddf93355de9541fbbdbd31fcea826a39 (127.10.96.193:43133)
I20260927 10:45:44.382112 19868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.193:39391
W20260927 10:45:44.486661 20059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:44.486909 20059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:44.486945 20059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:44.490799 20059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:44.490844 20059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:44.490893 20059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:44.495580 20059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:44.496542 20059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:44.497896 20059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:44.507081 20064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.507081 20067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.507113 20065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.507300 20059 server_base.cc:1034] running on GCE node
I20260927 10:45:44.507784 20059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:44.508389 20059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:44.509545 20059 hybrid_clock.cc:648] HybridClock initialized: now 1790505944509515 us; error 38 us; skew 500 ppm
I20260927 10:45:44.511312 20059 webserver.cc:460] Webserver started at http://127.10.96.194:33735/ using document root <none> and password file <none>
I20260927 10:45:44.511905 20059 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:44.511966 20059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:44.512195 20059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:44.514264 20059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "301e12025b564090a0e4559d441a5cf4"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "c9fe42cdea8e478da60fd1f1b0a7a454"
server_key_iv: "fc61a0231862266ed4ae720922b9362e"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.514815 20059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "301e12025b564090a0e4559d441a5cf4"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "c9fe42cdea8e478da60fd1f1b0a7a454"
server_key_iv: "fc61a0231862266ed4ae720922b9362e"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.518405 20059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:44.520823 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.522079 20059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:44.522208 20059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "301e12025b564090a0e4559d441a5cf4"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "c9fe42cdea8e478da60fd1f1b0a7a454"
server_key_iv: "fc61a0231862266ed4ae720922b9362e"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.522320 20059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:44.544979 20059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:44.545739 20059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:44.545938 20059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:44.546509 20059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:44.547513 20059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:44.547580 20059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.547649 20059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:44.547699 20059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.556119 20059 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:38239
I20260927 10:45:44.556142 20186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:38239 every 8 connection(s)
I20260927 10:45:44.556990 20059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:44.563275 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20059
I20260927 10:45:44.563390 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:44.563661 10627 external_mini_cluster.cc:1428] Setting key e3d468e7c0a46da78c25fbdb9a8d8e7e
I20260927 10:45:44.565636 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43351
--builtin_ntp_servers=127.10.96.212:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 10:45:44.566419 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.558695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36087 (local address 127.10.96.254:43351)
0927 10:45:44.558860 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:44.558865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:44.559909 (+ 1044us) server_negotiation.cc:407] Connection header received
0927 10:45:44.561207 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:44.561209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:44.561255 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:44.561338 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:44.562240 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.562761 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.564263 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.564558 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.564662 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.565879 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:44.565882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:44.565884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:44.565895 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:44.565912 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:44.565920 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:44.565956 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:44.566144 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 10:45:44.566217 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:45:44.567107 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.559118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43351 (local address 127.10.96.194:36087)
0927 10:45:44.559773 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:44.559812 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:44.561025 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:44.561565 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:44.561573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:44.561657 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:44.562124 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:44.562136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.562863 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.562865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:44.564141 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:44.564150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.564701 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.564704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:44.564898 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.565570 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:44.565572 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:44.565610 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:44.565670 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:44.566006 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:44.566012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:44.566016 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:45:44.566103 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:45:44.566309 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260927 10:45:44.568183 20187 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43351
I20260927 10:45:44.568415 20187 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:44.568935 20187 heartbeater.cc:511] Master 127.10.96.254:43351 requested a full tablet report, sending...
I20260927 10:45:44.570014 19868 ts_manager.cc:194] Registered new tserver with Master: 301e12025b564090a0e4559d441a5cf4 (127.10.96.194:38239)
I20260927 10:45:44.570573 19868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.194:36087
W20260927 10:45:44.674203 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:44.674475 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:44.674563 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:44.678541 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:44.678607 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:44.678686 20190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:44.683430 20190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:44.684460 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:44.685837 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:44.695094 20196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.695047 20198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.695047 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.695302 20190 server_base.cc:1034] running on GCE node
I20260927 10:45:44.695796 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:44.696455 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:44.697724 20190 hybrid_clock.cc:648] HybridClock initialized: now 1790505944697706 us; error 33 us; skew 500 ppm
I20260927 10:45:44.699507 20190 webserver.cc:460] Webserver started at http://127.10.96.195:44755/ using document root <none> and password file <none>
I20260927 10:45:44.700075 20190 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:44.700168 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:44.700402 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:44.702466 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "74d24f52cb8f483bbc65f0e0cc3da0c4"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "c3bc6edbf335ec0c2ba04d3c58d491ca"
server_key_iv: "3a70ce9b1017e0ea39e10397e9003517"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.703016 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "74d24f52cb8f483bbc65f0e0cc3da0c4"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "c3bc6edbf335ec0c2ba04d3c58d491ca"
server_key_iv: "3a70ce9b1017e0ea39e10397e9003517"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.706693 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:44.709012 20204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.710202 20190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:44.710330 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "74d24f52cb8f483bbc65f0e0cc3da0c4"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "c3bc6edbf335ec0c2ba04d3c58d491ca"
server_key_iv: "3a70ce9b1017e0ea39e10397e9003517"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.710446 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:44.741822 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:44.742551 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:44.742751 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:44.743355 20190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:44.744293 20190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:44.744364 20190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.744445 20190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:44.744498 20190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.752936 20190 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:35533
I20260927 10:45:44.752995 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:35533 every 8 connection(s)
I20260927 10:45:44.753888 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:44.762239 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20190
I20260927 10:45:44.762351 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:44.762667 10627 external_mini_cluster.cc:1428] Setting key e99644f1d91fc626018a671672febbe0
I20260927 10:45:44.763089 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.755709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:51729 (local address 127.10.96.254:43351)
0927 10:45:44.755857 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:44.755862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:44.756992 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 10:45:44.758345 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:44.758348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:44.758402 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:44.758480 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:44.759319 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.759828 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.760930 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.761141 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.761223 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.762532 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:44.762536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:44.762539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:44.762553 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:44.762576 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:44.762586 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:44.762663 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:44.762886 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:45:44.762948 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:45:44.764132 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.756171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43351 (local address 127.10.96.195:51729)
0927 10:45:44.756833 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:44.756871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:44.758159 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:44.758643 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:44.758650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:44.758733 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:44.759199 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:44.759211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.759934 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.759937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:44.760813 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:44.760821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:44.761322 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:44.761326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:44.761563 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:44.762226 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:44.762227 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:44.762267 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:44.762339 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:44.762676 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:44.762682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:44.762687 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:45:44.762864 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 10:45:44.763135 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
I20260927 10:45:44.765240 20318 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43351
I20260927 10:45:44.765483 20318 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:44.765998 20318 heartbeater.cc:511] Master 127.10.96.254:43351 requested a full tablet report, sending...
I20260927 10:45:44.766957 19868 ts_manager.cc:194] Registered new tserver with Master: 74d24f52cb8f483bbc65f0e0cc3da0c4 (127.10.96.195:35533)
I20260927 10:45:44.767515 19868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.195:51729
I20260927 10:45:44.776780 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:45:44.779783 19907 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:44.778813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.10.96.254:43351)
0927 10:45:44.778997 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:44.779001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:44.779118 (+ 117us) server_negotiation.cc:407] Connection header received
0927 10:45:44.779284 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:44.779287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:44.779337 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:44.779445 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:44.779679 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35216: BlockingRecv error: recv got EOF from 127.0.0.1:35216 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 10:45:44.780252 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19928
I20260927 10:45:44.785710 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20059
I20260927 10:45:44.791294 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20190
I20260927 10:45:44.796384 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 19837
2026-09-27T10:45:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (880 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T10:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:44Z Disabled control of system clock
I20260927 10:45:44.824671 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36153
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:36153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:45:44.933102 20329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:44.933362 20329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:44.933398 20329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:44.937233 20329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:44.937318 20329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:44.937335 20329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:44.937350 20329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:44.937366 20329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:44.942080 20329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:36153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:44.943092 20329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:44.944396 20329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:44.952584 20334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:44.952672 20337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.952687 20329 server_base.cc:1034] running on GCE node
W20260927 10:45:44.952584 20335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:44.953447 20329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:44.954563 20329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:44.955763 20329 hybrid_clock.cc:648] HybridClock initialized: now 1790505944955730 us; error 46 us; skew 500 ppm
I20260927 10:45:44.957651 20329 webserver.cc:460] Webserver started at http://127.10.96.254:38477/ using document root <none> and password file <none>
I20260927 10:45:44.958216 20329 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:44.958307 20329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:44.958511 20329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:44.960599 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "e49446e2130547b49a6f36bb8edd32ab"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "2afdc25011c5c08d59bb216beec2fa76"
server_key_iv: "dcf56e316a41427b17b0163692d9c2fa"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.961176 20329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "e49446e2130547b49a6f36bb8edd32ab"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "2afdc25011c5c08d59bb216beec2fa76"
server_key_iv: "dcf56e316a41427b17b0163692d9c2fa"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.964845 20329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:45:44.967298 20343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:44.968475 20329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:45:44.968614 20329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "e49446e2130547b49a6f36bb8edd32ab"
format_stamp: "Formatted at 2026-09-27 10:45:44 on dist-test-slave-wmg0"
server_key: "2afdc25011c5c08d59bb216beec2fa76"
server_key_iv: "dcf56e316a41427b17b0163692d9c2fa"
server_key_version: "encryptionkey@0"
I20260927 10:45:44.968727 20329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:44.973246 20329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:44.973881 20329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:44.974061 20329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:44.980633 20329 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:36153
I20260927 10:45:44.980690 20395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:36153 every 8 connection(s)
I20260927 10:45:44.981854 20329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:44.984942 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:44.990127 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20329
I20260927 10:45:44.990199 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab: Bootstrap starting.
I20260927 10:45:44.990309 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:44.990572 10627 external_mini_cluster.cc:1428] Setting key 00d7e87a3befeaa773910b41c4e8d05c
I20260927 10:45:44.992645 20396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:44.993359 20396 log.cc:824] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:44.995486 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab: No bootstrap required, opened a new log
I20260927 10:45:44.996088 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:44.991892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.10.96.254:36153)
0927 10:45:44.992492 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:44.992511 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:44.992544 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:45:44.993202 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:44.993219 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:44.993626 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:44.993898 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:44.994706 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:44.994711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:44.994725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:44.994736 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:44.994803 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:44.994825 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:44.994931 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:44.995019 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 10:45:44.995242 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260927 10:45:44.999504 20396 raft_consensus.cc:348] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49446e2130547b49a6f36bb8edd32ab" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36153 } }
I20260927 10:45:44.999799 20396 raft_consensus.cc:374] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:44.999858 20396 raft_consensus.cc:729] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49446e2130547b49a6f36bb8edd32ab, State: Initialized, Role: FOLLOWER
I20260927 10:45:45.000303 20396 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49446e2130547b49a6f36bb8edd32ab" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36153 } }
I20260927 10:45:45.000422 20396 raft_consensus.cc:388] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:45.000486 20396 raft_consensus.cc:482] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:45.000581 20396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:45.001591 20396 raft_consensus.cc:504] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49446e2130547b49a6f36bb8edd32ab" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36153 } }
I20260927 10:45:45.001955 20396 leader_election.cc:302] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e49446e2130547b49a6f36bb8edd32ab; no voters:
I20260927 10:45:45.002272 20396 leader_election.cc:288] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:45.002512 20401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:45.002730 20401 raft_consensus.cc:686] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [term 1 LEADER]: Becoming Leader. State: Replica: e49446e2130547b49a6f36bb8edd32ab, State: Running, Role: LEADER
I20260927 10:45:45.003017 20401 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49446e2130547b49a6f36bb8edd32ab" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36153 } }
I20260927 10:45:45.003914 20396 sys_catalog.cc:564] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:45.004712 20403 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49446e2130547b49a6f36bb8edd32ab. Latest consensus state: current_term: 1 leader_uuid: "e49446e2130547b49a6f36bb8edd32ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49446e2130547b49a6f36bb8edd32ab" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36153 } } }
I20260927 10:45:45.004853 20403 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:45.005192 20408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:45.005342 20402 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49446e2130547b49a6f36bb8edd32ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e49446e2130547b49a6f36bb8edd32ab" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36153 } } }
I20260927 10:45:45.005519 20402 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:45.008337 20408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:45.013226 20408 catalog_manager.cc:1347] Generated new cluster ID: 0c875f08b5d14e2b93e9f0457950941c
I20260927 10:45:45.013289 20408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:45.034492 20408 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:45.034801 20408 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:45.039534 20408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e49446e2130547b49a6f36bb8edd32ab: Generated new TSK 0
I20260927 10:45:45.040230 20408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:45:45.044847 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36153
--builtin_ntp_servers=127.10.96.212:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:45:45.155911 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:45.156160 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:45.156199 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:45.160285 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:45.160334 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:45.160425 20420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:45.165164 20420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:45.166308 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:45.167685 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:45.177136 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:45.177111 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:45.177162 20420 server_base.cc:1034] running on GCE node
W20260927 10:45:45.177112 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:45.177760 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:45.178390 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:45.179554 20420 hybrid_clock.cc:648] HybridClock initialized: now 1790505945179522 us; error 44 us; skew 500 ppm
I20260927 10:45:45.181289 20420 webserver.cc:460] Webserver started at http://127.10.96.193:46127/ using document root <none> and password file <none>
I20260927 10:45:45.182008 20420 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:45.182072 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:45.182282 20420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:45.184973 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "19e776df6b5d4708977f7f6b3e3be185"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "f0075d86eabe7c795d916d48260ac128"
server_key_iv: "784b5c25f8a580feeaa86faaa93db0ed"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.185778 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "19e776df6b5d4708977f7f6b3e3be185"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "f0075d86eabe7c795d916d48260ac128"
server_key_iv: "784b5c25f8a580feeaa86faaa93db0ed"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.189417 20420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:45.192039 20434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.193230 20420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:45.193408 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "19e776df6b5d4708977f7f6b3e3be185"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "f0075d86eabe7c795d916d48260ac128"
server_key_iv: "784b5c25f8a580feeaa86faaa93db0ed"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.193580 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:45.198135 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:45.198750 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:45.198953 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:45.199482 20420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:45.200414 20420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:45.200486 20420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.200554 20420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:45.200608 20420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.209753 20420 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39223
I20260927 10:45:45.209781 20547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39223 every 8 connection(s)
I20260927 10:45:45.210662 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:45.211020 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20420
I20260927 10:45:45.211139 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:45.211459 10627 external_mini_cluster.cc:1428] Setting key da2d77acc094565377bb47620c20eb02
I20260927 10:45:45.213696 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36153
--builtin_ntp_servers=127.10.96.212:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 10:45:45.217552 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:45.212608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:38397 (local address 127.10.96.254:36153)
0927 10:45:45.212741 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:45.212747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:45.214016 (+ 1269us) server_negotiation.cc:407] Connection header received
0927 10:45:45.215608 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:45.215611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:45.215645 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:45.215719 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:45.217016 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:45.217019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:45.217021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:45.217032 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:45.217059 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:45.217067 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:45.217145 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:45.217346 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:45:45.217384 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:45:45.218782 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:45.213084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36153 (local address 127.10.96.193:38397)
0927 10:45:45.213865 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:45.213911 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:45.215347 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:45.215928 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:45.215939 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:45.216088 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:45.216695 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:45.216700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:45.216756 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:45.216845 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:45.217159 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:45.217171 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:45.217181 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 10:45:45.217388 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 10:45:45.217676 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
I20260927 10:45:45.220062 20548 heartbeater.cc:347] Connected to a master server at 127.10.96.254:36153
I20260927 10:45:45.220290 20548 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:45.220703 20548 heartbeater.cc:511] Master 127.10.96.254:36153 requested a full tablet report, sending...
I20260927 10:45:45.222216 20360 ts_manager.cc:194] Registered new tserver with Master: 19e776df6b5d4708977f7f6b3e3be185 (127.10.96.193:39223)
W20260927 10:45:45.327450 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:45.327755 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:45.327831 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:45.331925 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:45.332010 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:45.332103 20551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:45.336764 20551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:45.337956 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:45.339368 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:45:45.349185 20551 server_base.cc:1034] running on GCE node
W20260927 10:45:45.349129 20557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:45.349131 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:45.349329 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:45.349761 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:45.350391 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:45.351583 20551 hybrid_clock.cc:648] HybridClock initialized: now 1790505945351571 us; error 30 us; skew 500 ppm
I20260927 10:45:45.353427 20551 webserver.cc:460] Webserver started at http://127.10.96.194:36747/ using document root <none> and password file <none>
I20260927 10:45:45.354068 20551 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:45.354162 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:45.354382 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:45.356410 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "a47ebd6082b64993ab7ab86d5ea22ad6"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "f45ccf1322d598a5a4868045906a394e"
server_key_iv: "7f294c6aa4d3a9fb76b82a4c4eeaea08"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.356980 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "a47ebd6082b64993ab7ab86d5ea22ad6"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "f45ccf1322d598a5a4868045906a394e"
server_key_iv: "7f294c6aa4d3a9fb76b82a4c4eeaea08"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.360803 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:45:45.363574 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.364871 20551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:45:45.365036 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a47ebd6082b64993ab7ab86d5ea22ad6"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "f45ccf1322d598a5a4868045906a394e"
server_key_iv: "7f294c6aa4d3a9fb76b82a4c4eeaea08"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.365171 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:45.369809 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:45.370414 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:45.370592 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:45.371109 20551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:45.371994 20551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:45.372063 20551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.372131 20551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:45.372167 20551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.380975 20551 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:38941
I20260927 10:45:45.380997 20678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:38941 every 8 connection(s)
I20260927 10:45:45.381948 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:45.388130 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:45.383802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:34117 (local address 127.10.96.254:36153)
0927 10:45:45.383944 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:45.383948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:45.384971 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 10:45:45.386389 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:45.386392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:45.386434 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:45.386539 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:45.387661 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:45.387665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:45.387667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:45.387678 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:45.387698 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:45.387707 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:45.387811 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:45.387949 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 10:45:45.387993 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:45:45.388965 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:45.384197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36153 (local address 127.10.96.194:34117)
0927 10:45:45.384827 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:45.384867 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:45.386167 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:45.386700 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:45.386706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:45.386795 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:45.387400 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:45.387401 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:45.387438 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:45.387505 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:45.387801 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:45.387808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:45.387814 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:45:45.387990 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 10:45:45.388218 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260927 10:45:45.390098 20679 heartbeater.cc:347] Connected to a master server at 127.10.96.254:36153
I20260927 10:45:45.390355 20679 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:45.390522 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20551
I20260927 10:45:45.390650 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:45.390801 20679 heartbeater.cc:511] Master 127.10.96.254:36153 requested a full tablet report, sending...
I20260927 10:45:45.390945 10627 external_mini_cluster.cc:1428] Setting key de76e53908ffb28f8eacaa6fba401364
I20260927 10:45:45.391831 20359 ts_manager.cc:194] Registered new tserver with Master: a47ebd6082b64993ab7ab86d5ea22ad6 (127.10.96.194:38941)
I20260927 10:45:45.392792 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36153
--builtin_ntp_servers=127.10.96.212:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 10:45:45.501502 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:45.501775 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:45.501839 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:45.506379 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:45.506438 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:45.506524 20682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:45.511394 20682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:36153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:45.512449 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:45.513882 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:45.523398 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:45.523394 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:45.523522 20682 server_base.cc:1034] running on GCE node
W20260927 10:45:45.523412 20687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:45.524072 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:45.524698 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:45.525898 20682 hybrid_clock.cc:648] HybridClock initialized: now 1790505945525883 us; error 28 us; skew 500 ppm
I20260927 10:45:45.527730 20682 webserver.cc:460] Webserver started at http://127.10.96.195:34871/ using document root <none> and password file <none>
I20260927 10:45:45.528301 20682 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:45.528395 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:45.528618 20682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:45.530663 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "73bf145b462940c5a95e0bf2ff16ac17"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "e846870704f26c7305a7feb1dd973fed"
server_key_iv: "23efa4a341b5d782174cdab5765d83d8"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.531224 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "73bf145b462940c5a95e0bf2ff16ac17"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "e846870704f26c7305a7feb1dd973fed"
server_key_iv: "23efa4a341b5d782174cdab5765d83d8"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.534790 20682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:45.537274 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.538568 20682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:45.538718 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "73bf145b462940c5a95e0bf2ff16ac17"
format_stamp: "Formatted at 2026-09-27 10:45:45 on dist-test-slave-wmg0"
server_key: "e846870704f26c7305a7feb1dd973fed"
server_key_iv: "23efa4a341b5d782174cdab5765d83d8"
server_key_version: "encryptionkey@0"
I20260927 10:45:45.538836 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:45.543519 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:45.544108 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:45.544287 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:45.544806 20682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:45.545753 20682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:45.545820 20682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.545910 20682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:45.545961 20682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:45.554328 20682 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:37697
I20260927 10:45:45.554351 20809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:37697 every 8 connection(s)
I20260927 10:45:45.555166 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:45.558622 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20682
I20260927 10:45:45.558733 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:45.558996 10627 external_mini_cluster.cc:1428] Setting key c26cad2d2ed846592f8dd49bf7bd15c7
I20260927 10:45:45.561312 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:45.556877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:45239 (local address 127.10.96.254:36153)
0927 10:45:45.557030 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:45.557035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:45.558146 (+ 1111us) server_negotiation.cc:407] Connection header received
0927 10:45:45.559564 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:45.559566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:45.559625 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:45.559711 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:45.560886 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:45.560896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:45.560899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:45:45.560909 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:45.560927 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:45:45.560935 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:45.561002 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:45.561172 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 10:45:45.561217 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:45:45.562150 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:45.557303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36153 (local address 127.10.96.195:45239)
0927 10:45:45.557999 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:45.558038 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:45.559338 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:45.559905 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:45.559924 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:45.560012 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:45.560624 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:45:45.560625 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:45.560664 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:45:45.560729 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:45.561018 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:45.561026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:45.561033 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 10:45:45.561184 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 10:45:45.561364 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":183,"threads_started":1}
I20260927 10:45:45.563098 20810 heartbeater.cc:347] Connected to a master server at 127.10.96.254:36153
I20260927 10:45:45.563310 20810 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:45.563691 20810 heartbeater.cc:511] Master 127.10.96.254:36153 requested a full tablet report, sending...
I20260927 10:45:45.564592 20359 ts_manager.cc:194] Registered new tserver with Master: 73bf145b462940c5a95e0bf2ff16ac17 (127.10.96.195:37697)
I20260927 10:45:45.572774 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 10:45:45.575529 20399 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:45:45.574689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.10.96.254:36153)
0927 10:45:45.574822 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:45.574827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:45.575010 (+ 183us) server_negotiation.cc:407] Connection header received
0927 10:45:45.575100 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:45.575102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:45.575132 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:45.575195 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:45.575380 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41652: BlockingRecv error: recv got EOF from 127.0.0.1:41652 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:45:45.575621 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20420
I20260927 10:45:45.580984 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20551
I20260927 10:45:45.586071 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20682
I20260927 10:45:45.590981 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20329
2026-09-27T10:45:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (794 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:45 dist-test-slave-wmg0 krb5kdc[20820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:45 dist-test-slave-wmg0 krb5kdc[20820](info): set up 2 sockets
Sep 27 10:45:45 dist-test-slave-wmg0 krb5kdc[20820](info): commencing operation
krb5kdc: starting...
W20260927 10:45:46.618479 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:46 dist-test-slave-wmg0 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:45:46.735929 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33025
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:46.845450 20833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:46.845855 20833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:46.845947 20833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:46.850065 20833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:45:46.850153 20833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:46.850189 20833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:46.850250 20833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:45:46.850304 20833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:45:46.855690 20833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:46.856783 20833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:46.857913 20833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:46.866439 20841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:46.866434 20839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:46.866436 20838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:46.866628 20833 server_base.cc:1034] running on GCE node
I20260927 10:45:46.867268 20833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:46.868361 20833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:46.869544 20833 hybrid_clock.cc:648] HybridClock initialized: now 1790505946869533 us; error 30 us; skew 500 ppm
Sep 27 10:45:46 dist-test-slave-wmg0 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505946, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:46.872241 20833 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:45:46.873124 20833 webserver.cc:460] Webserver started at http://127.10.96.254:33393/ using document root <none> and password file <none>
I20260927 10:45:46.873744 20833 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:46.873834 20833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:46.874048 20833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:46.876057 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6"
format_stamp: "Formatted at 2026-09-27 10:45:46 on dist-test-slave-wmg0"
server_key: "0b278c6137795267bb3a665de498662b"
server_key_iv: "d6ea112e6a48e83ce6fc806f0c6faf9b"
server_key_version: "encryptionkey@0"
I20260927 10:45:46.876616 20833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6"
format_stamp: "Formatted at 2026-09-27 10:45:46 on dist-test-slave-wmg0"
server_key: "0b278c6137795267bb3a665de498662b"
server_key_iv: "d6ea112e6a48e83ce6fc806f0c6faf9b"
server_key_version: "encryptionkey@0"
I20260927 10:45:46.880285 20833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:45:46.882781 20848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:46.883992 20833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:45:46.884145 20833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6"
format_stamp: "Formatted at 2026-09-27 10:45:46 on dist-test-slave-wmg0"
server_key: "0b278c6137795267bb3a665de498662b"
server_key_iv: "d6ea112e6a48e83ce6fc806f0c6faf9b"
server_key_version: "encryptionkey@0"
I20260927 10:45:46.884256 20833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:46.906289 20833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:46.909159 20833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:46.909373 20833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:46.916088 20833 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:33025
I20260927 10:45:46.916405 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:33025 every 8 connection(s)
I20260927 10:45:46.917066 20833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:45:46.919711 20901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:46.922032 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20833
I20260927 10:45:46.922152 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:45:46.922423 10627 external_mini_cluster.cc:1428] Setting key 210da64b1d53784d91104c77ceb24c01
I20260927 10:45:46.926003 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6: Bootstrap starting.
I20260927 10:45:46.928153 20901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:46.928864 20901 log.cc:824] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:46.930740 20901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6: No bootstrap required, opened a new log
Sep 27 10:45:46 dist-test-slave-wmg0 krb5kdc[20820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:46.933915 20901 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33025 } }
I20260927 10:45:46.934145 20901 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:46.934204 20901 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9b1b2c978b45b2a3f6f03a51ccdce6, State: Initialized, Role: FOLLOWER
I20260927 10:45:46.934659 20901 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33025 } }
I20260927 10:45:46.934809 20901 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:45:46.934886 20901 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:45:46.935020 20901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:46.935891 20901 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33025 } }
I20260927 10:45:46.936224 20901 leader_election.cc:302] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc9b1b2c978b45b2a3f6f03a51ccdce6; no voters:
I20260927 10:45:46.936592 20901 leader_election.cc:288] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:45:46.936659 20906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:46.936859 20906 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [term 1 LEADER]: Becoming Leader. State: Replica: bc9b1b2c978b45b2a3f6f03a51ccdce6, State: Running, Role: LEADER
I20260927 10:45:46.937157 20906 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33025 } }
I20260927 10:45:46.937481 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:46.923829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.10.96.254:33025)
0927 10:45:46.924295 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:46.924338 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:46.924378 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:45:46.925128 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:46.925147 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:46.925578 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:46.925915 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:46.926716 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:46.927698 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:46.928625 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:46.929153 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:46.929330 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:46.931971 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:46.931975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:46.931987 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:46.932013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:46.934375 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:46.934803 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:46.934809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:46.934815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:46.934986 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:46.935270 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:46.935275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:46.935277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:46.935658 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:46.935852 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:46.936153 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 10:45:46.936464 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":124,"threads_started":1}
I20260927 10:45:46.937997 20901 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:45:46.938493 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33025 } } }
I20260927 10:45:46.938603 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:46.938912 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc9b1b2c978b45b2a3f6f03a51ccdce6. Latest consensus state: current_term: 1 leader_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc9b1b2c978b45b2a3f6f03a51ccdce6" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33025 } } }
I20260927 10:45:46.939011 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:45:46.939504 20910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:45:46.942415 20910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:45:46.947703 20910 catalog_manager.cc:1347] Generated new cluster ID: 0a9eeccaaae64101b9828d071762a9e4
I20260927 10:45:46.947772 20910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:45:46.988394 20910 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:45:46.989908 20910 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:45:46.996824 20910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6: Generated new TSK 0
I20260927 10:45:46.997622 20910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:45:47.030301 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33025
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:47.140753 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:47.141042 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:47.141115 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:47.145117 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:47.145212 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:47.145308 20927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:45:47.150521 20927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:47.151556 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:47.152575 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:47.161842 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:47.161845 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:47.162190 20927 server_base.cc:1034] running on GCE node
W20260927 10:45:47.162267 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:47.162648 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:47.163286 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:47.164475 20927 hybrid_clock.cc:648] HybridClock initialized: now 1790505947164461 us; error 25 us; skew 500 ppm
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505947, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:47.167233 20927 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:45:47.168080 20927 webserver.cc:460] Webserver started at http://127.10.96.193:46183/ using document root <none> and password file <none>
I20260927 10:45:47.168637 20927 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:47.168725 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:47.168946 20927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:47.170990 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "42daf55f089a45708b0c7c2056f1e4b2"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "b1c5c32d39fbcbf2475ab35a5c163c48"
server_key_iv: "a89f82d89b951b0b387f463b0d981b4b"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.171533 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "42daf55f089a45708b0c7c2056f1e4b2"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "b1c5c32d39fbcbf2475ab35a5c163c48"
server_key_iv: "a89f82d89b951b0b387f463b0d981b4b"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.175138 20927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:45:47.177553 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.178716 20927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:45:47.178854 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "42daf55f089a45708b0c7c2056f1e4b2"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "b1c5c32d39fbcbf2475ab35a5c163c48"
server_key_iv: "a89f82d89b951b0b387f463b0d981b4b"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.178963 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:47.193321 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:47.196772 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:47.196974 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:47.197633 20927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:47.198437 20927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:47.198508 20927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.198577 20927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:47.198619 20927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.207221 20927 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39405
I20260927 10:45:47.207244 21055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39405 every 8 connection(s)
I20260927 10:45:47.208132 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:45:47.216975 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 20927
I20260927 10:45:47.217096 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:45:47.217439 10627 external_mini_cluster.cc:1428] Setting key 9befe90713d1e1d86d709970763c1662
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[20820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505947, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:47.222365 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.209933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:53419 (local address 127.10.96.254:33025)
0927 10:45:47.210060 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.210064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.210851 (+ 787us) server_negotiation.cc:407] Connection header received
0927 10:45:47.212229 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.212233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.212302 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.212402 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:47.213607 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.214141 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.215283 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.215511 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.215622 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.218290 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:47.218293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:47.218296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:47.218320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:47.219945 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.220567 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.220572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.220574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.220677 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.221039 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.221044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.221046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.221279 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:47.221417 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.221966 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.222107 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:47.222997 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.210270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33025 (local address 127.10.96.193:53419)
0927 10:45:47.210696 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.210737 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.212013 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.212578 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.212585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.212952 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:47.213420 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.213432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.214273 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.214277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.215150 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.215159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.215660 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.215664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.215836 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.216382 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:47.216401 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:47.218095 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:47.220061 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:47.220069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:47.220087 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:47.220454 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:47.220801 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:47.220804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:47.220805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:47.220945 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:47.221420 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:47.221424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:47.221725 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.221886 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.222129 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
I20260927 10:45:47.224246 21056 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33025
I20260927 10:45:47.224493 21056 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:47.224952 21056 heartbeater.cc:511] Master 127.10.96.254:33025 requested a full tablet report, sending...
I20260927 10:45:47.226404 20865 ts_manager.cc:194] Registered new tserver with Master: 42daf55f089a45708b0c7c2056f1e4b2 (127.10.96.193:39405)
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:45:47.227517 20865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:53419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:45:47.244182 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33025
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:47.355455 21061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:47.355752 21061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:47.355830 21061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:47.360059 21061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:47.360152 21061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:47.360240 21061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:45:47.365430 21061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:47.366557 21061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:47.367570 21061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:47.376902 21069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:47.376869 21067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:47.376869 21066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:47.377224 21061 server_base.cc:1034] running on GCE node
I20260927 10:45:47.377671 21061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:47.378234 21061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:47.379416 21061 hybrid_clock.cc:648] HybridClock initialized: now 1790505947379396 us; error 33 us; skew 500 ppm
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505947, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:47.382272 21061 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:45:47.383244 21061 webserver.cc:460] Webserver started at http://127.10.96.194:37683/ using document root <none> and password file <none>
I20260927 10:45:47.383792 21061 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:47.383880 21061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:47.384102 21061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:47.386193 21061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "7b81bccd57dc43fb9f028b420ec317a2"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "89ff8605109ec38e40402519b6fef8a5"
server_key_iv: "a24ba19a09e5944e831d4fedc52c1728"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.386840 21061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "7b81bccd57dc43fb9f028b420ec317a2"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "89ff8605109ec38e40402519b6fef8a5"
server_key_iv: "a24ba19a09e5944e831d4fedc52c1728"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.390583 21061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:45:47.392922 21076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.394148 21061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:45:47.394282 21061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "7b81bccd57dc43fb9f028b420ec317a2"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "89ff8605109ec38e40402519b6fef8a5"
server_key_iv: "a24ba19a09e5944e831d4fedc52c1728"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.394397 21061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:47.411489 21061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:47.414492 21061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:47.414700 21061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:47.415297 21061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:47.416107 21061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:47.416180 21061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.416250 21061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:47.416298 21061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.425576 21061 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:35097
I20260927 10:45:47.425592 21189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:35097 every 8 connection(s)
I20260927 10:45:47.426502 21061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:45:47.430455 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 21061
I20260927 10:45:47.430572 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:45:47.430871 10627 external_mini_cluster.cc:1428] Setting key a3d5ac2f3ab4e9a46a6a0f339cd4d28f
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[20820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505947, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:45:47.441758 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.428301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:48277 (local address 127.10.96.254:33025)
0927 10:45:47.428465 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.428469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.429313 (+ 844us) server_negotiation.cc:407] Connection header received
0927 10:45:47.431068 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.431071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.431115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.431182 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:47.432571 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.433083 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.434535 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.434922 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.435040 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.437900 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:47.437904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:47.437909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:47.437937 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:47.439523 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.440078 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.440086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.440087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.440157 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.440527 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.440530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.440532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.440715 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:47.440819 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.441526 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.441639 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 10:45:47.442711 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.428715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33025 (local address 127.10.96.194:48277)
0927 10:45:47.429155 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.429198 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.430839 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.431366 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.431373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.431808 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:47.432428 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.432448 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.433208 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.433215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.434414 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.434422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.435073 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.435080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.435343 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.435975 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:47.436001 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:47.437730 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:47.439637 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:47.439642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:47.439656 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:47.439932 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:47.440256 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:47.440259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:47.440261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:47.440428 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:47.440825 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:47.440834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:47.441196 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.441405 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.441719 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:45:47.444051 21190 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33025
I20260927 10:45:47.444290 21190 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:47.444778 21190 heartbeater.cc:511] Master 127.10.96.254:33025 requested a full tablet report, sending...
I20260927 10:45:47.445819 20865 ts_manager.cc:194] Registered new tserver with Master: 7b81bccd57dc43fb9f028b420ec317a2 (127.10.96.194:35097)
I20260927 10:45:47.446408 20865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:48277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:45:47.456368 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33025
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:45:47.566547 21195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:45:47.566793 21195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:45:47.566834 21195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:45:47.570760 21195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:45:47.570811 21195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:45:47.570866 21195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:45:47.576167 21195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:45:47.577098 21195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:45:47.578132 21195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:45:47.587401 21201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:47.587361 21200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:45:47.587387 21203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:45:47.587657 21195 server_base.cc:1034] running on GCE node
I20260927 10:45:47.588104 21195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:45:47.588760 21195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:45:47.589946 21195 hybrid_clock.cc:648] HybridClock initialized: now 1790505947589919 us; error 49 us; skew 500 ppm
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[20820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505947, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:45:47.592711 21195 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:45:47.593729 21195 webserver.cc:460] Webserver started at http://127.10.96.195:43811/ using document root <none> and password file <none>
I20260927 10:45:47.594281 21195 fs_manager.cc:359] Metadata directory not provided
I20260927 10:45:47.594379 21195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:45:47.594607 21195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:45:47.596935 21195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "14ea0b03578940939ab16ee7538c35cf"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "9897889e577036dadc47df881b57ae88"
server_key_iv: "87278f9eb6bf6ec1a056e0a7bcbe05e8"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.597594 21195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "14ea0b03578940939ab16ee7538c35cf"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "9897889e577036dadc47df881b57ae88"
server_key_iv: "87278f9eb6bf6ec1a056e0a7bcbe05e8"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.601248 21195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:45:47.603895 21210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.605137 21195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:45:47.605276 21195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "14ea0b03578940939ab16ee7538c35cf"
format_stamp: "Formatted at 2026-09-27 10:45:47 on dist-test-slave-wmg0"
server_key: "9897889e577036dadc47df881b57ae88"
server_key_iv: "87278f9eb6bf6ec1a056e0a7bcbe05e8"
server_key_version: "encryptionkey@0"
I20260927 10:45:47.605420 21195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:45:47.641898 21195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:45:47.645076 21195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:45:47.645300 21195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:45:47.645980 21195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:45:47.646816 21195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:45:47.646886 21195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.646952 21195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:45:47.647006 21195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:45:47.655437 21195 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:34903
I20260927 10:45:47.655470 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:34903 every 8 connection(s)
I20260927 10:45:47.656351 21195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:45:47.662865 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 21195
I20260927 10:45:47.662999 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:45:47.663301 10627 external_mini_cluster.cc:1428] Setting key b2bda2b47d5a1cf0f66df5a2317d84a2
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[20820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505947, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:45:47.670276 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.658130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56053 (local address 127.10.96.254:33025)
0927 10:45:47.658275 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.658278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.659106 (+ 828us) server_negotiation.cc:407] Connection header received
0927 10:45:47.660482 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.660486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.660543 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.660646 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:47.661847 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.662390 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.663958 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.664234 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.664364 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.666857 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:47.666861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:47.666864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:47.666894 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:47.668176 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.668674 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.668680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.668682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.668768 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.669096 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.669101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.669103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.669278 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:47.669392 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.669992 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.670110 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:45:47.671051 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.658520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33025 (local address 127.10.96.195:56053)
0927 10:45:47.658959 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.659001 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.660269 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.660817 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.660824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.661172 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:45:47.661710 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.661725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.662515 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.662518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.663820 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.663829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.664390 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.664394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.664563 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.665156 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:45:47.665176 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:45:47.666701 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:45:47.668295 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:47.668301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:47.668316 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:47.668570 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:47.668862 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:45:47.668865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:45:47.668867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:45:47.669002 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:45:47.669390 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:45:47.669396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:45:47.669723 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.669940 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.670255 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260927 10:45:47.672031 21324 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33025
I20260927 10:45:47.672266 21324 heartbeater.cc:464] Registering TS with master...
I20260927 10:45:47.672727 21324 heartbeater.cc:511] Master 127.10.96.254:33025 requested a full tablet report, sending...
I20260927 10:45:47.673691 20865 ts_manager.cc:194] Registered new tserver with Master: 14ea0b03578940939ab16ee7538c35cf (127.10.96.195:34903)
I20260927 10:45:47.674193 20865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:56053
I20260927 10:45:47.676964 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:45:47.685889 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.678870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.10.96.254:33025)
0927 10:45:47.679008 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.679012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.679182 (+ 170us) server_negotiation.cc:407] Connection header received
0927 10:45:47.679261 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.679263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.679298 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.679365 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:45:47.679976 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.680467 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.681219 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.681426 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.681585 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.682988 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:45:47.682991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:45:47.682993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:45:47.683012 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:45:47.684272 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.684679 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.684682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.684683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.684759 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:45:47.684993 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:45:47.684996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:45:47.684997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:45:47.685191 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:45:47.685289 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.685595 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.685688 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:45:47.688889 20865 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:45:47.691795 20865 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:45:47.706382 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.700141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39405 (local address 127.0.0.1:54606)
0927 10:45:47.700955 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.700969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.701051 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.701806 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.701811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.701834 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:47.701971 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.701978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.703436 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.703439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.704660 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.704670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.705843 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.705847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.705951 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.705989 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.706076 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.706203 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":203,"thread_start_us":84,"threads_started":1}
I20260927 10:45:47.706696 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.700225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.10.96.194:35097)
0927 10:45:47.701627 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.701638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.701661 (+ 23us) server_negotiation.cc:407] Connection header received
0927 10:45:47.701736 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.701741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.701916 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.702066 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:47.702488 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.703545 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.704883 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.705897 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.706000 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.706143 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.706454 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.706557 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1303,"thread_start_us":182,"threads_started":1}
I20260927 10:45:47.706763 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.700829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:34903 (local address 127.0.0.1:47556)
0927 10:45:47.701231 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.701243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.701317 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.702059 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.702063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.702085 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:47.702168 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.702174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.703188 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.703194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.704667 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.704672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.706034 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.706037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.706185 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.706205 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.706278 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.706334 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":83,"thread_start_us":89,"threads_started":1}
I20260927 10:45:47.706945 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.700359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.10.96.193:39405)
0927 10:45:47.701276 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.701293 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.701319 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:45:47.701398 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.701405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.701643 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.701812 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:47.702105 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.703301 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.704813 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.705672 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.706470 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.706622 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.706679 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.706774 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":99,"threads_started":1}
I20260927 10:45:47.707188 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.700134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:35097 (local address 127.0.0.1:40134)
0927 10:45:47.700827 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.700875 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.701002 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.702235 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.702241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.702258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:47.702361 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.702367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.703709 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.703715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.704727 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.704738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.706010 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.706014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.706179 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.706202 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.706319 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.706393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":31,"thread_start_us":81,"threads_started":1}
I20260927 10:45:47.707578 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.700985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.10.96.195:34903)
0927 10:45:47.701662 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.701681 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.701699 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:45:47.701759 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.701763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.701922 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.702066 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:47.702258 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.703057 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.704802 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.705672 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.705810 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.705976 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.706361 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.706722 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":139,"threads_started":1}
I20260927 10:45:47.709894 20990 tablet_service.cc:1467] Processing CreateTablet for tablet f73ea00ca16b46468ae591e1a3872797 (DEFAULT_TABLE table=test-table [id=6cc1896084304029a84a53d12a3d4d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:47.709993 21258 tablet_service.cc:1467] Processing CreateTablet for tablet f73ea00ca16b46468ae591e1a3872797 (DEFAULT_TABLE table=test-table [id=6cc1896084304029a84a53d12a3d4d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:47.709892 21124 tablet_service.cc:1467] Processing CreateTablet for tablet f73ea00ca16b46468ae591e1a3872797 (DEFAULT_TABLE table=test-table [id=6cc1896084304029a84a53d12a3d4d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:45:47.710950 21124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73ea00ca16b46468ae591e1a3872797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:47.710955 21258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73ea00ca16b46468ae591e1a3872797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:47.710950 20990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f73ea00ca16b46468ae591e1a3872797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:45:47.716831 21341 tablet_bootstrap.cc:492] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2: Bootstrap starting.
I20260927 10:45:47.718775 21342 tablet_bootstrap.cc:492] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: Bootstrap starting.
I20260927 10:45:47.718861 21343 tablet_bootstrap.cc:492] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Bootstrap starting.
I20260927 10:45:47.719224 21341 tablet_bootstrap.cc:654] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:47.720278 21341 log.cc:824] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:47.720938 21343 tablet_bootstrap.cc:654] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:47.721421 21342 tablet_bootstrap.cc:654] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: Neither blocks nor log segments found. Creating new log.
I20260927 10:45:47.722107 21341 tablet_bootstrap.cc:492] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2: No bootstrap required, opened a new log
I20260927 10:45:47.722260 21343 log.cc:824] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:47.722270 21342 log.cc:824] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:45:47.722308 21341 ts_tablet_manager.cc:1397] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:45:47.724205 21343 tablet_bootstrap.cc:492] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: No bootstrap required, opened a new log
I20260927 10:45:47.724387 21343 ts_tablet_manager.cc:1397] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:45:47.724505 21342 tablet_bootstrap.cc:492] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: No bootstrap required, opened a new log
I20260927 10:45:47.724680 21342 ts_tablet_manager.cc:1397] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:45:47.725956 21341 raft_consensus.cc:348] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.726190 21341 raft_consensus.cc:374] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:47.726246 21341 raft_consensus.cc:729] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42daf55f089a45708b0c7c2056f1e4b2, State: Initialized, Role: FOLLOWER
I20260927 10:45:47.726672 21341 consensus_queue.cc:260] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.727239 21056 heartbeater.cc:503] Master 127.10.96.254:33025 was elected leader, sending a full tablet report...
I20260927 10:45:47.727609 21341 ts_tablet_manager.cc:1428] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:45:47.728233 21342 raft_consensus.cc:348] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.728217 21343 raft_consensus.cc:348] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.728461 21343 raft_consensus.cc:374] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:47.728461 21342 raft_consensus.cc:374] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:45:47.728508 21342 raft_consensus.cc:729] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b81bccd57dc43fb9f028b420ec317a2, State: Initialized, Role: FOLLOWER
I20260927 10:45:47.728508 21343 raft_consensus.cc:729] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14ea0b03578940939ab16ee7538c35cf, State: Initialized, Role: FOLLOWER
I20260927 10:45:47.728920 21343 consensus_queue.cc:260] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.728920 21342 consensus_queue.cc:260] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.729812 21324 heartbeater.cc:503] Master 127.10.96.254:33025 was elected leader, sending a full tablet report...
I20260927 10:45:47.729863 21343 ts_tablet_manager.cc:1428] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260927 10:45:47.729863 21342 ts_tablet_manager.cc:1428] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:45:47.730252 21190 heartbeater.cc:503] Master 127.10.96.254:33025 was elected leader, sending a full tablet report...
I20260927 10:45:47.750248 21349 raft_consensus.cc:482] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:45:47.750430 21349 raft_consensus.cc:504] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.751493 21349 leader_election.cc:288] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b81bccd57dc43fb9f028b420ec317a2 (127.10.96.194:35097), 42daf55f089a45708b0c7c2056f1e4b2 (127.10.96.193:39405)
I20260927 10:45:47.756451 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.751845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:52881 (local address 127.10.96.193:39405)
0927 10:45:47.752004 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.752011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.752277 (+ 266us) server_negotiation.cc:407] Connection header received
0927 10:45:47.752359 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.752361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.752397 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.752456 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:47.752720 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.753444 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.755198 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.755797 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.755921 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.755973 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.756251 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.756322 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:45:47.756451 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.751836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:53703 (local address 127.10.96.194:35097)
0927 10:45:47.751997 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.752002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.752019 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:45:47.752071 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.752077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.752133 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.752203 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:45:47.752556 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.753331 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.755198 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.755797 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.755921 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.755973 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.756251 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.756323 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:45:47.756482 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.751699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:35097 (local address 127.10.96.195:53703)
0927 10:45:47.751820 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.751835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.751921 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.752254 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.752257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.752277 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:47.752410 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.752415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.753448 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.753451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.755057 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.755068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.755945 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.755949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.756103 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.756139 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.756224 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.756344 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":208}
I20260927 10:45:47.756496 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.751718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39405 (local address 127.10.96.195:52881)
0927 10:45:47.752155 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.752165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 10:45:47.752273 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:45:47.752497 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:45:47.752501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:45:47.752520 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:45:47.752618 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.752622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.753544 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.753547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.755057 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:45:47.755068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.755945 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.755949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:45:47.756099 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.756151 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 10:45:47.756224 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:45:47.756316 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":258,"thread_start_us":135,"threads_started":1}
I20260927 10:45:47.757206 21144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73ea00ca16b46468ae591e1a3872797" candidate_uuid: "14ea0b03578940939ab16ee7538c35cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b81bccd57dc43fb9f028b420ec317a2" is_pre_election: true
I20260927 10:45:47.757385 21010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73ea00ca16b46468ae591e1a3872797" candidate_uuid: "14ea0b03578940939ab16ee7538c35cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42daf55f089a45708b0c7c2056f1e4b2" is_pre_election: true
I20260927 10:45:47.757568 21144 raft_consensus.cc:2454] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea0b03578940939ab16ee7538c35cf in term 0.
I20260927 10:45:47.757745 21010 raft_consensus.cc:2454] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14ea0b03578940939ab16ee7538c35cf in term 0.
I20260927 10:45:47.758118 21213 leader_election.cc:302] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea0b03578940939ab16ee7538c35cf, 7b81bccd57dc43fb9f028b420ec317a2; no voters:
I20260927 10:45:47.758410 21349 raft_consensus.cc:2781] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:45:47.758502 21349 raft_consensus.cc:482] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:45:47.758562 21349 raft_consensus.cc:3037] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:47.759546 21349 raft_consensus.cc:504] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.759892 21349 leader_election.cc:288] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [CANDIDATE]: Term 1 election: Requested vote from peers 7b81bccd57dc43fb9f028b420ec317a2 (127.10.96.194:35097), 42daf55f089a45708b0c7c2056f1e4b2 (127.10.96.193:39405)
I20260927 10:45:47.760226 21010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73ea00ca16b46468ae591e1a3872797" candidate_uuid: "14ea0b03578940939ab16ee7538c35cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42daf55f089a45708b0c7c2056f1e4b2"
I20260927 10:45:47.760371 21010 raft_consensus.cc:3037] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:47.761405 21010 raft_consensus.cc:2454] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea0b03578940939ab16ee7538c35cf in term 1.
I20260927 10:45:47.761616 21144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f73ea00ca16b46468ae591e1a3872797" candidate_uuid: "14ea0b03578940939ab16ee7538c35cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b81bccd57dc43fb9f028b420ec317a2"
I20260927 10:45:47.761734 21144 raft_consensus.cc:3037] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:45:47.762220 21212 leader_election.cc:302] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14ea0b03578940939ab16ee7538c35cf, 42daf55f089a45708b0c7c2056f1e4b2; no voters:
I20260927 10:45:47.762420 21349 raft_consensus.cc:2781] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:45:47.762671 21349 raft_consensus.cc:686] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 1 LEADER]: Becoming Leader. State: Replica: 14ea0b03578940939ab16ee7538c35cf, State: Running, Role: LEADER
I20260927 10:45:47.763046 21349 consensus_queue.cc:237] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } }
I20260927 10:45:47.763094 21144 raft_consensus.cc:2454] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14ea0b03578940939ab16ee7538c35cf in term 1.
I20260927 10:45:47.766063 20862 catalog_manager.cc:5495] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 14ea0b03578940939ab16ee7538c35cf (127.10.96.195). New cstate: current_term: 1 leader_uuid: "14ea0b03578940939ab16ee7538c35cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14ea0b03578940939ab16ee7538c35cf" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 34903 } health_report { overall_health: HEALTHY } } }
I20260927 10:45:47.786548 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.782815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47560 (local address 127.10.96.195:34903)
0927 10:45:47.782955 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:45:47.782962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:45:47.782980 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:45:47.783173 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:45:47.783176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:45:47.783239 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:45:47.783385 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:45:47.783616 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.784151 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.785022 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:45:47.785257 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:45:47.785383 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:45:47.785731 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:45:47.786133 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:45:47.786266 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 10:45:47.786325 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 10:45:47.786387 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:45:47.792801 21010 raft_consensus.cc:1261] T f73ea00ca16b46468ae591e1a3872797 P 42daf55f089a45708b0c7c2056f1e4b2 [term 1 FOLLOWER]: Refusing update from remote peer 14ea0b03578940939ab16ee7538c35cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:47.792847 21144 raft_consensus.cc:1261] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 1 FOLLOWER]: Refusing update from remote peer 14ea0b03578940939ab16ee7538c35cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:45:47.793535 21349 consensus_queue.cc:1035] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "42daf55f089a45708b0c7c2056f1e4b2" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:47.793731 21351 consensus_queue.cc:1035] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b81bccd57dc43fb9f028b420ec317a2" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:45:47.800020 21354 mvcc.cc:205] Tried to move back new op lower bound from 7333912362151288832 to 7333912362039988224. Current Snapshot: MvccSnapshot[applied={T|T < 7333912362151288832}]
I20260927 10:45:47.801939 21355 mvcc.cc:205] Tried to move back new op lower bound from 7333912362151288832 to 7333912362039988224. Current Snapshot: MvccSnapshot[applied={T|T < 7333912362151288832}]
I20260927 10:45:47.802204 21356 mvcc.cc:205] Tried to move back new op lower bound from 7333912362151288832 to 7333912362039988224. Current Snapshot: MvccSnapshot[applied={T|T < 7333912362151288832}]
I20260927 10:45:47.810395 20862 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:47.810593 20862 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:45:47.813035 20862 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc9b1b2c978b45b2a3f6f03a51ccdce6: Sending DeleteTablet for 3 replicas of tablet f73ea00ca16b46468ae591e1a3872797
I20260927 10:45:47.813855 21124 tablet_service.cc:1514] Processing DeleteTablet for tablet f73ea00ca16b46468ae591e1a3872797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:47 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:40134
I20260927 10:45:47.813865 20990 tablet_service.cc:1514] Processing DeleteTablet for tablet f73ea00ca16b46468ae591e1a3872797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:47 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54606
I20260927 10:45:47.814013 21258 tablet_service.cc:1514] Processing DeleteTablet for tablet f73ea00ca16b46468ae591e1a3872797 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:45:47 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:47556
I20260927 10:45:47.814517 21364 tablet_replica.cc:330] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: stopping tablet replica
I20260927 10:45:47.814520 21365 tablet_replica.cc:330] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: stopping tablet replica
I20260927 10:45:47.814975 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20927
I20260927 10:45:47.815018 21365 raft_consensus.cc:2229] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:45:47.816511 21365 raft_consensus.cc:2258] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:47.818053 21365 ts_tablet_manager.cc:1905] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:47.814834 21364 raft_consensus.cc:2229] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:45:47.821761 21365 ts_tablet_manager.cc:1918] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:45:47.821888 21364 raft_consensus.cc:2258] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:45:47.822006 21365 log.cc:1196] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/f73ea00ca16b46468ae591e1a3872797
I20260927 10:45:47.822566 21365 ts_tablet_manager.cc:1939] T f73ea00ca16b46468ae591e1a3872797 P 14ea0b03578940939ab16ee7538c35cf: Deleting consensus metadata
W20260927 10:45:47.823204 20850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:39405 (error 108)
I20260927 10:45:47.823450 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 21061
I20260927 10:45:47.823508 21364 ts_tablet_manager.cc:1905] T f73ea00ca16b46468ae591e1a3872797 P 7b81bccd57dc43fb9f028b420ec317a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:45:47.826984 20849 catalog_manager.cc:4841] TS 14ea0b03578940939ab16ee7538c35cf (127.10.96.195:34903): tablet f73ea00ca16b46468ae591e1a3872797 (table test-table [id=6cc1896084304029a84a53d12a3d4d02]) successfully deleted
I20260927 10:45:47.829888 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.829527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:35097 (local address 127.0.0.1:40142)
0927 10:45:47.829707 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.829782 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:35097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 10:45:47.830083 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:45:47.829819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39405 (local address 127.0.0.1:54610)
0927 10:45:47.829955 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 10:45:47.830010 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:45:47.830116 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 21195
W20260927 10:45:47.830842 20851 catalog_manager.cc:4584] TS 7b81bccd57dc43fb9f028b420ec317a2 (127.10.96.194:35097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f73ea00ca16b46468ae591e1a3872797: Network error: Client connection negotiation failed: client connection to 127.10.96.194:35097: connect: Connection refused (error 111)
I20260927 10:45:47.835769 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 20833
2026-09-27T10:45:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2246 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[21369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[21369](info): set up 2 sockets
Sep 27 10:45:47 dist-test-slave-wmg0 krb5kdc[21369](info): commencing operation
krb5kdc: starting...
W20260927 10:45:48.874534 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:45:48 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:45:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:45:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/rangeradmin_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/rangeradmin_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/rangerlookup_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/rangerlookup_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
I20260927 10:45:49.029953 10627 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKx0Sgr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/postgres -l logfile start
2026-09-27 10:45:51.374 UTC [21399] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:45:51.374 UTC [21399] LOG: listening on IPv4 address "127.10.96.212", port 60871
2026-09-27 10:45:51.377 UTC [21399] LOG: listening on Unix socket "/tmp/.s.PGSQL.60871"
2026-09-27 10:45:51.399 UTC [21402] LOG: database system was shut down at 2026-09-27 10:45:50 UTC
2026-09-27 10:45:51.403 UTC [21399] LOG: database system is ready to accept connections
I20260927 10:45:52.342263 10627 mini_postgres.cc:96] Postgres bound to 60871
2026-09-27 10:45:52.348 UTC [21410] FATAL: database "slave" does not exist
127.10.96.212:60871 - accepting connections
I20260927 10:45:52.349269 10627 mini_ranger.cc:161] Starting Ranger...
I20260927 10:45:52.363889 10627 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:45:52.797628 10627 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:45:52.797726 10627 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-admin
2026-09-27 10:45:53,072 [I] DB FLAVOR :POSTGRES
2026-09-27 10:45:53,073 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:45:53,073 [I] Checking connection..
2026-09-27 10:45:53,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:45:53,318 [I] Checking connection passed.
2026-09-27 10:45:53,319 [I] --------- Verifying version history table ---------
2026-09-27 10:45:53,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:45:53,584 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:45:53,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:45:53,809 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:45:53,809 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:45:53,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:45:54.062 UTC [21474] WARNING: there is no transaction in progress
2026-09-27 10:45:54,068 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:45:54,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:45:54,310 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:45:54,310 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:45:54,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:45:54,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:45:54,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wmg0.c.gcp-upstream.internal','N') ;"
2026-09-27 10:45:55,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:45:55,243 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:45:55,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:45:55,506 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:45:55,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:45:55,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:45:56,006 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:45:56,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:45:56.923 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:56.938 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:56.954 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:56.970 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:56.986 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.003 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.057 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.064 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.077 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.087 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.098 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.112 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.121 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.127 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.134 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57.141 UTC [21582] WARNING: there is no transaction in progress
2026-09-27 10:45:57,752 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:45:57,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:45:57,994 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:45:57,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:45:58,253 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:45:58,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:45:58,509 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:45:58,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:45:58,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:45:58,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wmg0.c.gcp-upstream.internal';"
2026-09-27 10:45:59,225 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:45:59,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:45:59,468 [I] DB_PATCHES have already been applied
I20260927 10:45:59.474112 10627 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKx0Sgr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:45:59.474174 10627 mini_ranger.cc:193] Using host: 127.10.96.212
I20260927 10:45:59.477427 10627 mini_ranger.cc:227] Ranger admin URL: http://127.10.96.212:50805
Sep 27, 2026 10:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:46:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:46:00.579231 10627 mini_ranger.cc:160] Time spent starting Ranger: real 8.230s user 0.000s sys 0.010s
Sep 27, 2026 10:46:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:46:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
Sep 27, 2026 10:46:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:46:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:46:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:46:22.046852 10627 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/rangerkms_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/rangerkms_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.96.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.96.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:46:22 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:46:22.133720 10627 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:46:22.144320 10627 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:46:22.921721 10627 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:46:22.921834 10627 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms
I20260927 10:46:22.921947 10627 mini_ranger_kms.cc:227] Using postgres at 127.10.96.212:60871
2026-09-27 10:46:23,106 [I] DB FLAVOR :POSTGRES
2026-09-27 10:46:23,106 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:46:23,106 [I] Checking connection
2026-09-27 10:46:23,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:46:23,340 [I] connection success
2026-09-27 10:46:23,340 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:46:23,340 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:46:23,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:46:23,573 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:46:23,573 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:46:23,573 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:46:23,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:60871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:46:23,830 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:46:24.255978 10627 mini_ranger_kms.cc:310] Created kms service
I20260927 10:46:24.456663 10627 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:46:24.502526 10627 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:46:24 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505982, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:46:24.890595 10627 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:46:24.890738 10627 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:46:24.890762 10627 mini_ranger_kms.cc:240] Using host: 127.10.96.212
I20260927 10:46:24.894359 10627 mini_ranger_kms.cc:276] Ranger KMS PID: 21762
I20260927 10:46:24.894505 10627 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.96.212:38455
Sep 27, 2026 10:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:46:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:46:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:46:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:46:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:46:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 10:46:27.146270 10627 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.013s user 0.000s sys 0.169s
I20260927 10:46:27.146409 10627 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:46:27.146513 10627 mini_ranger_kms.cc:401] 127.10.96.212:38455/kms/v1/keys
Sep 27 10:46:27 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505987, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.96.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 10:46:27 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505987, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.96.212@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
Sep 27 10:46:27 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505987, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.96.212@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
Sep 27 10:46:30 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:46:30.145201 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35537
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:35537
--ranger_config_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:38455/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:30.256875 21813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:30.257198 21813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:30.257303 21813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:30.261574 21813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:46:30.261660 21813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:30.261698 21813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:30.261731 21813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:46:30.261760 21813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:46:30.267982 21813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.96.212:38455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:35537
--ranger_config_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:46:30.269111 21813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:30.270267 21813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:30.280018 21818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:30.280035 21821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:30.280042 21819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:30.280509 21813 server_base.cc:1034] running on GCE node
I20260927 10:46:30.281172 21813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:30.282589 21813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:30.283790 21813 hybrid_clock.cc:648] HybridClock initialized: now 1790505990283760 us; error 41 us; skew 500 ppm
Sep 27 10:46:30 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505990, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:30.290655 21813 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:46:30.291690 21813 webserver.cc:460] Webserver started at http://127.10.96.254:38461/ using document root <none> and password file <none>
I20260927 10:46:30.292263 21813 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:30.292352 21813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:30.292572 21813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:30 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505990, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:46:30.354569 21813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "06c66d5344ff41a9b48f49a49e81948f"
format_stamp: "Formatted at 2026-09-27 10:46:30 on dist-test-slave-wmg0"
server_key: "c24217ce3c8f4a45a98c27bfa5362761"
server_key_iv: "c7efe7264d0906c422708026186e200a"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:30.355362 21813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "06c66d5344ff41a9b48f49a49e81948f"
format_stamp: "Formatted at 2026-09-27 10:46:30 on dist-test-slave-wmg0"
server_key: "c24217ce3c8f4a45a98c27bfa5362761"
server_key_iv: "c7efe7264d0906c422708026186e200a"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:30.360569 21813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 10:46:30.378784 21828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:30.380509 21813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:46:30.380661 21813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "06c66d5344ff41a9b48f49a49e81948f"
format_stamp: "Formatted at 2026-09-27 10:46:30 on dist-test-slave-wmg0"
server_key: "c24217ce3c8f4a45a98c27bfa5362761"
server_key_iv: "c7efe7264d0906c422708026186e200a"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:30.380800 21813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:30.396030 21813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:30.399174 21813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:30.399379 21813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:30.407979 21813 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:35537
I20260927 10:46:30.408038 21880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:35537 every 8 connection(s)
I20260927 10:46:30.408946 21813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:46:30.412422 21881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:30.412987 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 21813
I20260927 10:46:30.413136 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:46:30.413427 10627 external_mini_cluster.cc:1428] Setting key e8683de416a5606f83a60d958f1c0d4b
I20260927 10:46:30.420265 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f: Bootstrap starting.
Sep 27 10:46:30 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:46:30.423118 21881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:30.424039 21881 log.cc:824] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:30.426378 21881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f: No bootstrap required, opened a new log
I20260927 10:46:30.427915 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:30.414977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.10.96.254:35537)
0927 10:46:30.415381 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:30.415396 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:30.415429 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:46:30.416080 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:30.416096 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:30.416406 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:30.416747 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:30.417589 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:30.418560 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:30.419419 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:30.419880 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:30.420061 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:30.422571 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:30.422577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:30.422590 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:30.422614 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:30.425042 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:30.425516 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:30.425522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:30.425532 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:30.425630 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:30.425929 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:30.425933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:30.425935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:30.426304 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:30.426567 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:30.426775 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 10:46:30.427159 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260927 10:46:30.430519 21881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c66d5344ff41a9b48f49a49e81948f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35537 } }
I20260927 10:46:30.430804 21881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:30.430851 21881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c66d5344ff41a9b48f49a49e81948f, State: Initialized, Role: FOLLOWER
I20260927 10:46:30.431308 21881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c66d5344ff41a9b48f49a49e81948f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35537 } }
I20260927 10:46:30.431468 21881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:46:30.431524 21881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:46:30.431630 21881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:30.433262 21881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c66d5344ff41a9b48f49a49e81948f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35537 } }
I20260927 10:46:30.433667 21881 leader_election.cc:302] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c66d5344ff41a9b48f49a49e81948f; no voters:
I20260927 10:46:30.434064 21881 leader_election.cc:288] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:46:30.434383 21886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:30.434643 21886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [term 1 LEADER]: Becoming Leader. State: Replica: 06c66d5344ff41a9b48f49a49e81948f, State: Running, Role: LEADER
I20260927 10:46:30.434985 21886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c66d5344ff41a9b48f49a49e81948f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35537 } }
I20260927 10:46:30.435395 21881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:46:30.436641 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c66d5344ff41a9b48f49a49e81948f. Latest consensus state: current_term: 1 leader_uuid: "06c66d5344ff41a9b48f49a49e81948f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c66d5344ff41a9b48f49a49e81948f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35537 } } }
I20260927 10:46:30.436767 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:30.437091 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c66d5344ff41a9b48f49a49e81948f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c66d5344ff41a9b48f49a49e81948f" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35537 } } }
I20260927 10:46:30.437182 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f [sys.catalog]: This master's current role is: LEADER
I20260927 10:46:30.437711 21881 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:46:30.438705 21895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:46:30.444357 21895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:46:30.454026 21895 catalog_manager.cc:1347] Generated new cluster ID: d5a54e7ba08c49e8ab3cae53e6a3d25a
I20260927 10:46:30.454089 21895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:46:30.475297 21895 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:46:30.476631 21895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:46:30.491446 21895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f: Generated new TSK 0
I20260927 10:46:30.492395 21895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:46:31.257452 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:38455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35537
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:46:31.432984 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:31.433401 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:31.433497 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:31.439554 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:31.439658 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:31.439760 21926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:46:31.446095 21926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.96.212:38455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:46:31.447304 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:31.448395 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:31.457953 21934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:31.458186 21933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:31.457933 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:31.459311 21926 server_base.cc:1034] running on GCE node
I20260927 10:46:31.459784 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:31.460459 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:31.461632 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790505991461601 us; error 50 us; skew 500 ppm
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:31.465216 21926 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:46:31.466464 21926 webserver.cc:460] Webserver started at http://127.10.96.193:44141/ using document root <none> and password file <none>
I20260927 10:46:31.467221 21926 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:31.467310 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:31.467581 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:46:31.487670 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "1e0e6de1c26c49319a6c041fb86c1623"
format_stamp: "Formatted at 2026-09-27 10:46:31 on dist-test-slave-wmg0"
server_key: "5047b711fd1dc5918b5cc423e42fa757"
server_key_iv: "92fb360d7df7b607b8edf3b7662624b9"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:31.488305 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "1e0e6de1c26c49319a6c041fb86c1623"
format_stamp: "Formatted at 2026-09-27 10:46:31 on dist-test-slave-wmg0"
server_key: "5047b711fd1dc5918b5cc423e42fa757"
server_key_iv: "92fb360d7df7b607b8edf3b7662624b9"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:31.492422 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:46:31.512719 21943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:31.514575 21926 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:46:31.514710 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "1e0e6de1c26c49319a6c041fb86c1623"
format_stamp: "Formatted at 2026-09-27 10:46:31 on dist-test-slave-wmg0"
server_key: "5047b711fd1dc5918b5cc423e42fa757"
server_key_iv: "92fb360d7df7b607b8edf3b7662624b9"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:31.514860 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:31.533581 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:31.537533 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:31.537719 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:31.538347 21926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:31.539371 21926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:31.539463 21926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:31.539525 21926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:31.539556 21926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:31.553845 21926 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:42407
I20260927 10:46:31.553967 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:42407 every 8 connection(s)
I20260927 10:46:31.554821 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:46:31.557359 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 21926
I20260927 10:46:31.557535 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:46:31.557843 10627 external_mini_cluster.cc:1428] Setting key 7a6d9d3bd737efbba176ee09ce058d7d
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:46:31.578935 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.556937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:49573 (local address 127.10.96.254:35537)
0927 10:46:31.562942 (+ 6005us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:31.562947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:31.564891 (+ 1944us) server_negotiation.cc:407] Connection header received
0927 10:46:31.564959 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:31.564962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:31.565020 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:31.565119 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:31.567359 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.568271 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.570002 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.570398 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.570519 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:31.573894 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:31.573898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:31.573900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:31.573926 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:31.576031 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:31.576787 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:31.576793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:31.576796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:31.576894 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:31.577342 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:31.577348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:31.577351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:31.577680 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:31.577780 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:31.578566 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 10:46:31.578727 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5899,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:46:31.580721 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.557503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35537 (local address 127.10.96.193:49573)
0927 10:46:31.563285 (+ 5782us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:31.563371 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:31.564533 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:31.565926 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:31.565937 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:31.566340 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:31.567184 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:31.567198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.568463 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.568469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:31.569837 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:31.569853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.570712 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.570716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:31.570926 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:31.571659 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:31.571689 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:31.573651 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:31.576194 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:31.576205 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:31.576229 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:31.576656 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:31.577007 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:31.577010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:31.577012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:31.577224 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:31.577840 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:31.577847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:31.578224 (+ 377us) client_negotiation.cc:769] Sending connection context
0927 10:46:31.579188 (+ 964us) client_negotiation.cc:240] Negotiation successful
0927 10:46:31.579608 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5499,"thread_start_us":176,"threads_started":1}
I20260927 10:46:31.582136 22057 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35537
I20260927 10:46:31.582381 22057 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:31.582986 22057 heartbeater.cc:511] Master 127.10.96.254:35537 requested a full tablet report, sending...
I20260927 10:46:31.584626 21845 ts_manager.cc:194] Registered new tserver with Master: 1e0e6de1c26c49319a6c041fb86c1623 (127.10.96.193:42407)
I20260927 10:46:31.586126 21845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:49573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:46:31.614332 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:38455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35537
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:46:31.773677 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:31.774060 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:31.774132 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:31.780371 22065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:31.780462 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:31.780551 22065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:46:31.788550 22065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.96.212:38455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:46:31.789999 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:31.791347 22065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:31.803886 22071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:31.803990 22072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:31.803890 22074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:31.804661 22065 server_base.cc:1034] running on GCE node
I20260927 10:46:31.805214 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:31.806023 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:31.807229 22065 hybrid_clock.cc:648] HybridClock initialized: now 1790505991807196 us; error 47 us; skew 500 ppm
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:31.810985 22065 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:46:31.812142 22065 webserver.cc:460] Webserver started at http://127.10.96.194:32797/ using document root <none> and password file <none>
I20260927 10:46:31.812908 22065 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:31.813009 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:31.813714 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:46:31.832049 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "b790b0ec06514c908828d21080fa5e2d"
format_stamp: "Formatted at 2026-09-27 10:46:31 on dist-test-slave-wmg0"
server_key: "7090dab48f204b9e05b5ed592489e920"
server_key_iv: "569ddceaad3bde414112afb73ee3e1b5"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:31.832856 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "b790b0ec06514c908828d21080fa5e2d"
format_stamp: "Formatted at 2026-09-27 10:46:31 on dist-test-slave-wmg0"
server_key: "7090dab48f204b9e05b5ed592489e920"
server_key_iv: "569ddceaad3bde414112afb73ee3e1b5"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:31.838896 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 10:46:31.865283 22081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:31.867125 22065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:46:31.867347 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "b790b0ec06514c908828d21080fa5e2d"
format_stamp: "Formatted at 2026-09-27 10:46:31 on dist-test-slave-wmg0"
server_key: "7090dab48f204b9e05b5ed592489e920"
server_key_iv: "569ddceaad3bde414112afb73ee3e1b5"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:31.867587 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:31.898779 22065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:31.902195 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:31.902407 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:31.903028 22065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:31.903983 22065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:31.904048 22065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:31.904127 22065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:31.904160 22065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:31.917815 22065 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:41297
I20260927 10:46:31.918993 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:46:31.925110 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 22065
I20260927 10:46:31.925261 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:46:31.925236 22194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:41297 every 8 connection(s)
I20260927 10:46:31.925611 10627 external_mini_cluster.cc:1428] Setting key 5abaf09ea50a61b42f9fc7730ea3c30a
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Sep 27 10:46:31 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:46:31.950352 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.929330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:53997 (local address 127.10.96.254:35537)
0927 10:46:31.929451 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:31.929478 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:31.930583 (+ 1105us) server_negotiation.cc:407] Connection header received
0927 10:46:31.931853 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:31.931857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:31.931927 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:31.931981 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:31.933747 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.934573 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.936399 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.936773 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.936880 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:31.945505 (+ 8625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:31.945511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:31.945514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:31.945548 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:31.947426 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:31.948206 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:31.948211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:31.948213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:31.948319 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:31.948812 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:31.948816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:31.948818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:31.949077 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:31.949171 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:31.950013 (+ 842us) server_negotiation.cc:299] Negotiation successful
0927 10:46:31.950174 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:46:31.952364 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:31.929138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35537 (local address 127.10.96.194:53997)
0927 10:46:31.930419 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:31.930482 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:31.931588 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:31.932412 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:31.932422 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:31.932802 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:31.933569 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:31.933592 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.934741 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.934744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:31.936254 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:31.936266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:31.937153 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:31.937157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:31.937363 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:31.941172 (+ 3809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:31.941194 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:31.945192 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:31.947610 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:31.947617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:31.947638 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:31.948032 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:31.948455 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:31.948458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:31.948461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:31.948688 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:31.949252 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:31.949260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:31.949673 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 10:46:31.950783 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0927 10:46:31.951207 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":161,"threads_started":1}
I20260927 10:46:31.954093 22195 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35537
I20260927 10:46:31.954350 22195 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:31.954965 22195 heartbeater.cc:511] Master 127.10.96.254:35537 requested a full tablet report, sending...
I20260927 10:46:31.956239 21845 ts_manager.cc:194] Registered new tserver with Master: b790b0ec06514c908828d21080fa5e2d (127.10.96.194:41297)
I20260927 10:46:31.957016 21845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:53997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:46:31.965196 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:38455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35537
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:46:32 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
W20260927 10:46:32.111725 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:46:32.112217 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:46:32.112326 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:46:32.118669 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:46:32.118832 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:46:32.119009 22200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:46:32.126926 22200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.96.212:38455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:35537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:46:32.128592 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:46:32.130046 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:46:32.143587 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:32.143604 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:46:32.144127 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:46:32.145309 22200 server_base.cc:1034] running on GCE node
I20260927 10:46:32.145857 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:46:32.146687 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:46:32.147910 22200 hybrid_clock.cc:648] HybridClock initialized: now 1790505992147873 us; error 52 us; skew 500 ppm
Sep 27 10:46:32 dist-test-slave-wmg0 krb5kdc[21369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:46:32.151705 22200 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:46:32.152796 22200 webserver.cc:460] Webserver started at http://127.10.96.195:32853/ using document root <none> and password file <none>
I20260927 10:46:32.153503 22200 fs_manager.cc:359] Metadata directory not provided
I20260927 10:46:32.153577 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:46:32.153801 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:46:32 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:46:32.187987 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "0b827f3f36c54454a0cc0d6a0b844f00"
format_stamp: "Formatted at 2026-09-27 10:46:32 on dist-test-slave-wmg0"
server_key: "f388549cdd43f577581b5f1b2703bf7b"
server_key_iv: "fd8010d5797c8d7c30043676f9ff4eaf"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:32.188956 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "0b827f3f36c54454a0cc0d6a0b844f00"
format_stamp: "Formatted at 2026-09-27 10:46:32 on dist-test-slave-wmg0"
server_key: "f388549cdd43f577581b5f1b2703bf7b"
server_key_iv: "fd8010d5797c8d7c30043676f9ff4eaf"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:32.202564 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260927 10:46:32.229233 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:32.231885 22200 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.000s
I20260927 10:46:32.232071 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "0b827f3f36c54454a0cc0d6a0b844f00"
format_stamp: "Formatted at 2026-09-27 10:46:32 on dist-test-slave-wmg0"
server_key: "f388549cdd43f577581b5f1b2703bf7b"
server_key_iv: "fd8010d5797c8d7c30043676f9ff4eaf"
server_key_version: "kuduclusterkey@0"
I20260927 10:46:32.232235 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:46:32.271958 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:46:32.275648 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:46:32.275957 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:46:32.276690 22200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:46:32.277957 22200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:46:32.278060 22200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:32.278149 22200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:46:32.278205 22200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:46:32.288748 22200 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:35701
I20260927 10:46:32.289940 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:46:32.291667 22328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:35701 every 8 connection(s)
I20260927 10:46:32.296298 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 22200
I20260927 10:46:32.296407 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:46:32.296669 10627 external_mini_cluster.cc:1428] Setting key d9a27eb6f769df5d723175310d299551
Sep 27 10:46:32 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:46:32.314273 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.294989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:36913 (local address 127.10.96.254:35537)
0927 10:46:32.295111 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.295115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.295904 (+ 789us) server_negotiation.cc:407] Connection header received
0927 10:46:32.299925 (+ 4021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.299929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.299981 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.300033 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:32.301790 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.302622 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.304310 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.304671 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.304744 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.308095 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:32.308098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:32.308102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:32.308128 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:32.311772 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:32.312431 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:32.312434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:32.312436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:32.312531 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:32.312945 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:32.312948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:32.312949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:32.313161 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:32.313235 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.313979 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.314114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:46:32.316218 22331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.293920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35537 (local address 127.10.96.195:36913)
0927 10:46:32.295750 (+ 1830us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.295800 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.299673 (+ 3873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.300465 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:32.300488 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:32.300847 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:46:32.301552 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.301573 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.302781 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.302784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.304172 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.304183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.304945 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.304949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.305145 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.305896 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:46:32.305924 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:46:32.307902 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:46:32.311926 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:32.311932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:32.311952 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:32.312316 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:32.312645 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:46:32.312648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:46:32.312650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:46:32.312833 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:46:32.313298 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:46:32.313306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:46:32.313686 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 10:46:32.314755 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0927 10:46:32.315125 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1620,"thread_start_us":137,"threads_started":1}
I20260927 10:46:32.317759 22329 heartbeater.cc:347] Connected to a master server at 127.10.96.254:35537
I20260927 10:46:32.317996 22329 heartbeater.cc:464] Registering TS with master...
I20260927 10:46:32.318562 22329 heartbeater.cc:511] Master 127.10.96.254:35537 requested a full tablet report, sending...
I20260927 10:46:32.319734 21845 ts_manager.cc:194] Registered new tserver with Master: 0b827f3f36c54454a0cc0d6a0b844f00 (127.10.96.195:35701)
I20260927 10:46:32.320497 21845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:36913
I20260927 10:46:32.322590 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:46:32.337160 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.327034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.10.96.254:35537)
0927 10:46:32.327140 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.327145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.327698 (+ 553us) server_negotiation.cc:407] Connection header received
0927 10:46:32.327886 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.327890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.327935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.327984 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:46:32.328950 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.329826 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.331032 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.331364 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.331434 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.333241 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:46:32.333245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:46:32.333248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:46:32.333273 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:46:32.335038 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:32.335584 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:32.335588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:32.335590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:32.335667 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:46:32.336011 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:46:32.336015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:46:32.336018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:46:32.336264 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:46:32.336367 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.336874 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.336996 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:46:32.341245 21845 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:46:32.345928 21845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:46:32.375224 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.361919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:42407 (local address 127.0.0.1:41640)
0927 10:46:32.367124 (+ 5205us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.367186 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.367314 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.368530 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:32.368537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:32.368561 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:32.369584 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.369595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.371030 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.371033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.373293 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.373303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.374688 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.374692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.374801 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.374836 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:46:32.374942 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:46:32.375065 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5000,"thread_start_us":142,"threads_started":1}
I20260927 10:46:32.376865 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.362831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.10.96.193:42407)
0927 10:46:32.367815 (+ 4984us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.367822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.367847 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:46:32.367927 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.367933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.368156 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.368305 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:32.369738 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.370900 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.373427 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.374374 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.374475 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.374631 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.375843 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.376725 (+ 882us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4869,"thread_start_us":67,"threads_started":1}
I20260927 10:46:32.380610 21988 tablet_service.cc:1467] Processing CreateTablet for tablet ed443293cc44489ab9a0653b54bf6882 (DEFAULT_TABLE table=test-table [id=a1b2d05d591e48ee9a7bd01a26b7888a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:32.381985 21988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed443293cc44489ab9a0653b54bf6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:32.383620 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.364020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:41297 (local address 127.0.0.1:45678)
0927 10:46:32.369172 (+ 5152us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.369190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.369301 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.370063 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:32.370066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:32.370083 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:32.370181 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.370187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.372141 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.372144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.378635 (+ 6491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.378646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.383212 (+ 4566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.383216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.383321 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.383349 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:46:32.383398 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 10:46:32.383475 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5075,"thread_start_us":93,"threads_started":1}
I20260927 10:46:32.389364 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.364413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.10.96.194:41297)
0927 10:46:32.367605 (+ 3192us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.367621 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.369412 (+ 1791us) server_negotiation.cc:407] Connection header received
0927 10:46:32.369532 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.369538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.369751 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.369901 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:32.370311 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.371473 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.379744 (+ 8271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.381571 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.384089 (+ 2518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.388281 (+ 4192us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.388345 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.389216 (+ 871us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3082,"thread_start_us":114,"threads_started":1}
I20260927 10:46:32.393322 22112 tablet_service.cc:1467] Processing CreateTablet for tablet ed443293cc44489ab9a0653b54bf6882 (DEFAULT_TABLE table=test-table [id=a1b2d05d591e48ee9a7bd01a26b7888a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:32.394516 22112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed443293cc44489ab9a0653b54bf6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:32.400826 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.365122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35701 (local address 127.0.0.1:45072)
0927 10:46:32.369606 (+ 4484us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.369621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.369700 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.378428 (+ 8728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:32.378431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:32.378452 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:32.378577 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.378586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.391576 (+ 12990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.391580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.392770 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.392782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.400350 (+ 7568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.400355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.400484 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.400518 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:46:32.400576 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 10:46:32.400667 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4425,"thread_start_us":84,"threads_started":1}
I20260927 10:46:32.402530 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.365579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.10.96.195:35701)
0927 10:46:32.377844 (+ 12265us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.377856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.377884 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:46:32.378008 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.378014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.378249 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.381694 (+ 3445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:32.381847 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.391431 (+ 9584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.394790 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.395623 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.401162 (+ 5539us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.402230 (+ 1068us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.402287 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.402393 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12140,"thread_start_us":130,"threads_started":1}
I20260927 10:46:32.403643 22348 tablet_bootstrap.cc:492] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: Bootstrap starting.
I20260927 10:46:32.406158 22348 tablet_bootstrap.cc:654] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:32.406009 22263 tablet_service.cc:1467] Processing CreateTablet for tablet ed443293cc44489ab9a0653b54bf6882 (DEFAULT_TABLE table=test-table [id=a1b2d05d591e48ee9a7bd01a26b7888a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:46:32.407068 22348 log.cc:824] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:32.407189 22263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed443293cc44489ab9a0653b54bf6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:46:32.403926 22350 tablet_bootstrap.cc:492] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: Bootstrap starting.
I20260927 10:46:32.415309 22352 tablet_bootstrap.cc:492] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Bootstrap starting.
I20260927 10:46:32.417367 22350 tablet_bootstrap.cc:654] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:32.418008 22352 tablet_bootstrap.cc:654] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Neither blocks nor log segments found. Creating new log.
I20260927 10:46:32.418856 22352 log.cc:824] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:32.420917 22352 tablet_bootstrap.cc:492] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: No bootstrap required, opened a new log
I20260927 10:46:32.421098 22352 ts_tablet_manager.cc:1397] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:46:32.424906 22352 raft_consensus.cc:348] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.425082 22350 log.cc:824] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:46:32.425177 22352 raft_consensus.cc:374] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:32.425233 22352 raft_consensus.cc:729] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b827f3f36c54454a0cc0d6a0b844f00, State: Initialized, Role: FOLLOWER
I20260927 10:46:32.425513 22348 tablet_bootstrap.cc:492] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: No bootstrap required, opened a new log
I20260927 10:46:32.425709 22348 ts_tablet_manager.cc:1397] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260927 10:46:32.426076 22352 consensus_queue.cc:260] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.427423 22352 ts_tablet_manager.cc:1428] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:46:32.427829 22329 heartbeater.cc:503] Master 127.10.96.254:35537 was elected leader, sending a full tablet report...
I20260927 10:46:32.430048 22348 raft_consensus.cc:348] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.430343 22348 raft_consensus.cc:374] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:32.430394 22348 raft_consensus.cc:729] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0e6de1c26c49319a6c041fb86c1623, State: Initialized, Role: FOLLOWER
I20260927 10:46:32.430846 22348 consensus_queue.cc:260] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.431910 22348 ts_tablet_manager.cc:1428] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:46:32.435745 22350 tablet_bootstrap.cc:492] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: No bootstrap required, opened a new log
I20260927 10:46:32.436069 22350 ts_tablet_manager.cc:1397] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.004s
Sep 27 10:46:32 dist-test-slave-wmg0 krb5kdc[21369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:46:32.441094 22057 heartbeater.cc:503] Master 127.10.96.254:35537 was elected leader, sending a full tablet report...
I20260927 10:46:32.447947 22350 raft_consensus.cc:348] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.448304 22350 raft_consensus.cc:374] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:46:32.448380 22350 raft_consensus.cc:729] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b790b0ec06514c908828d21080fa5e2d, State: Initialized, Role: FOLLOWER
I20260927 10:46:32.448925 22350 consensus_queue.cc:260] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.449744 22195 heartbeater.cc:503] Master 127.10.96.254:35537 was elected leader, sending a full tablet report...
I20260927 10:46:32.450068 22350 ts_tablet_manager.cc:1428] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260927 10:46:32.498374 22356 raft_consensus.cc:482] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:46:32.498638 22356 raft_consensus.cc:504] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.500053 22356 leader_election.cc:288] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b790b0ec06514c908828d21080fa5e2d (127.10.96.194:41297), 0b827f3f36c54454a0cc0d6a0b844f00 (127.10.96.195:35701)
I20260927 10:46:32.506711 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.501322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:49735 (local address 127.10.96.195:35701)
0927 10:46:32.501480 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.501485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.502188 (+ 703us) server_negotiation.cc:407] Connection header received
0927 10:46:32.502477 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.502480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.502525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.502573 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:32.502986 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.503765 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.505390 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.506017 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.506110 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.506160 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.506510 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.506584 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:46:32.507305 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.500676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35701 (local address 127.10.96.193:49735)
0927 10:46:32.502104 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.502118 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.502336 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.502705 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:32.502707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:32.502724 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:32.502859 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.502865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.503947 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.503950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.505248 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.505259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.506265 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.506268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.506362 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.506383 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:46:32.507090 (+ 707us) client_negotiation.cc:240] Negotiation successful
0927 10:46:32.507166 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":91,"threads_started":1}
I20260927 10:46:32.508080 22283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed443293cc44489ab9a0653b54bf6882" candidate_uuid: "1e0e6de1c26c49319a6c041fb86c1623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" is_pre_election: true
I20260927 10:46:32.508389 22283 raft_consensus.cc:2454] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0e6de1c26c49319a6c041fb86c1623 in term 0.
I20260927 10:46:32.508988 21945 leader_election.cc:302] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b827f3f36c54454a0cc0d6a0b844f00, 1e0e6de1c26c49319a6c041fb86c1623; no voters:
I20260927 10:46:32.509279 22356 raft_consensus.cc:2781] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:46:32.509342 22356 raft_consensus.cc:482] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:46:32.509378 22356 raft_consensus.cc:3037] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:32.510697 22356 raft_consensus.cc:504] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.511045 22356 leader_election.cc:288] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [CANDIDATE]: Term 1 election: Requested vote from peers b790b0ec06514c908828d21080fa5e2d (127.10.96.194:41297), 0b827f3f36c54454a0cc0d6a0b844f00 (127.10.96.195:35701)
I20260927 10:46:32.511668 22283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed443293cc44489ab9a0653b54bf6882" candidate_uuid: "1e0e6de1c26c49319a6c041fb86c1623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b827f3f36c54454a0cc0d6a0b844f00"
I20260927 10:46:32.511786 22283 raft_consensus.cc:3037] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:32.513844 22283 raft_consensus.cc:2454] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0e6de1c26c49319a6c041fb86c1623 in term 1.
I20260927 10:46:32.514271 21945 leader_election.cc:302] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b827f3f36c54454a0cc0d6a0b844f00, 1e0e6de1c26c49319a6c041fb86c1623; no voters:
I20260927 10:46:32.514516 22356 raft_consensus.cc:2781] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:46:32.514748 22356 raft_consensus.cc:686] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [term 1 LEADER]: Becoming Leader. State: Replica: 1e0e6de1c26c49319a6c041fb86c1623, State: Running, Role: LEADER
I20260927 10:46:32.515108 22356 consensus_queue.cc:237] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } }
I20260927 10:46:32.519419 21845 catalog_manager.cc:5495] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e0e6de1c26c49319a6c041fb86c1623 (127.10.96.193). New cstate: current_term: 1 leader_uuid: "1e0e6de1c26c49319a6c041fb86c1623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e0e6de1c26c49319a6c041fb86c1623" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 42407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 } health_report { overall_health: UNKNOWN } } }
I20260927 10:46:32.526230 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.501151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:53339 (local address 127.10.96.194:41297)
0927 10:46:32.501499 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.501504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.501521 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:46:32.501790 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.501793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.501846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.501900 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:46:32.514894 (+ 12994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.515718 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.522843 (+ 7125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.523495 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.523632 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.523690 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.525995 (+ 2305us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.526094 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260927 10:46:32.528594 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.500230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:41297 (local address 127.10.96.193:53339)
0927 10:46:32.501240 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.501256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.501523 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.514590 (+ 13067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:46:32.514593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:46:32.514618 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:46:32.514744 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.514749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.515867 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.515870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.517149 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:46:32.517159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.525066 (+ 7907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.525070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:46:32.525326 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.525354 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 10:46:32.528169 (+ 2815us) client_negotiation.cc:240] Negotiation successful
0927 10:46:32.528405 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":125,"threads_started":1}
I20260927 10:46:32.529606 22136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed443293cc44489ab9a0653b54bf6882" candidate_uuid: "1e0e6de1c26c49319a6c041fb86c1623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b790b0ec06514c908828d21080fa5e2d" is_pre_election: true
I20260927 10:46:32.529940 22136 raft_consensus.cc:2454] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0e6de1c26c49319a6c041fb86c1623 in term 0.
I20260927 10:46:32.530323 22131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed443293cc44489ab9a0653b54bf6882" candidate_uuid: "1e0e6de1c26c49319a6c041fb86c1623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b790b0ec06514c908828d21080fa5e2d"
I20260927 10:46:32.530428 22131 raft_consensus.cc:3037] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:46:32.531765 22131 raft_consensus.cc:2454] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0e6de1c26c49319a6c041fb86c1623 in term 1.
W20260927 10:46:32.545686 22330 tablet.cc:2369] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:46:32.557979 22058 tablet.cc:2369] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:46:32.585038 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.573922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.10.96.193:42407)
0927 10:46:32.574425 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 10:46:32.574431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:46:32.574447 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:46:32.574549 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:46:32.574553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:46:32.574623 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:46:32.574734 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:46:32.575154 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.575933 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.578317 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:46:32.578687 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:46:32.578793 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:46:32.583159 (+ 4366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:46:32.584145 (+ 986us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:46:32.584282 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:46:32.584806 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 10:46:32.584878 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413}
I20260927 10:46:32.598383 22283 raft_consensus.cc:1261] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 1 FOLLOWER]: Refusing update from remote peer 1e0e6de1c26c49319a6c041fb86c1623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:32.599344 22361 consensus_queue.cc:1035] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b827f3f36c54454a0cc0d6a0b844f00" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:32.607486 22131 raft_consensus.cc:1261] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 1 FOLLOWER]: Refusing update from remote peer 1e0e6de1c26c49319a6c041fb86c1623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:46:32.608280 22361 consensus_queue.cc:1035] T ed443293cc44489ab9a0653b54bf6882 P 1e0e6de1c26c49319a6c041fb86c1623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b790b0ec06514c908828d21080fa5e2d" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 41297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:46:32.623538 22364 mvcc.cc:205] Tried to move back new op lower bound from 7333912545672839168 to 7333912545346048000. Current Snapshot: MvccSnapshot[applied={T|T < 7333912545672839168}]
I20260927 10:46:32.633678 22367 mvcc.cc:205] Tried to move back new op lower bound from 7333912545672839168 to 7333912545346048000. Current Snapshot: MvccSnapshot[applied={T|T < 7333912545672839168}]
I20260927 10:46:32.635046 22365 mvcc.cc:205] Tried to move back new op lower bound from 7333912545672839168 to 7333912545346048000. Current Snapshot: MvccSnapshot[applied={T|T < 7333912545672839168}]
I20260927 10:46:32.645866 21845 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:32.646063 21845 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:46:32.648761 21845 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06c66d5344ff41a9b48f49a49e81948f: Sending DeleteTablet for 3 replicas of tablet ed443293cc44489ab9a0653b54bf6882
I20260927 10:46:32.650090 21988 tablet_service.cc:1514] Processing DeleteTablet for tablet ed443293cc44489ab9a0653b54bf6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:32 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:41640
I20260927 10:46:32.650619 22263 tablet_service.cc:1514] Processing DeleteTablet for tablet ed443293cc44489ab9a0653b54bf6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:32 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:45072
I20260927 10:46:32.651530 22112 tablet_service.cc:1514] Processing DeleteTablet for tablet ed443293cc44489ab9a0653b54bf6882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:46:32 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:45678
I20260927 10:46:32.652189 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 21926
I20260927 10:46:32.653666 22375 tablet_replica.cc:330] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: stopping tablet replica
I20260927 10:46:32.654040 22375 raft_consensus.cc:2229] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:32.654322 22375 raft_consensus.cc:2258] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:32.661995 22374 tablet_replica.cc:330] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: stopping tablet replica
I20260927 10:46:32.662459 22374 raft_consensus.cc:2229] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:46:32.662914 22374 raft_consensus.cc:2258] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:46:32.665059 22374 ts_tablet_manager.cc:1905] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:32.665663 22375 ts_tablet_manager.cc:1905] T ed443293cc44489ab9a0653b54bf6882 P b790b0ec06514c908828d21080fa5e2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:46:32.668141 22374 ts_tablet_manager.cc:1918] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 10:46:32.669800 21832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:42407 (error 108)
I20260927 10:46:32.670650 22374 log.cc:1196] T ed443293cc44489ab9a0653b54bf6882 P 0b827f3f36c54454a0cc0d6a0b844f00: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/ed443293cc44489ab9a0653b54bf6882
I20260927 10:46:32.670938 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 22065
I20260927 10:46:32.678653 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.678039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:42407 (local address 127.0.0.1:41648)
0927 10:46:32.678240 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.678553 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:42407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260927 10:46:32.678959 21832 catalog_manager.cc:4584] TS 1e0e6de1c26c49319a6c041fb86c1623 (127.10.96.193:42407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed443293cc44489ab9a0653b54bf6882: Network error: Client connection negotiation failed: client connection to 127.10.96.193:42407: connect: Connection refused (error 111)
I20260927 10:46:32.682250 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:46:32.681957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:41297 (local address 127.0.0.1:45686)
0927 10:46:32.682118 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.682169 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:41297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 10:46:32.682816 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 22200
W20260927 10:46:32.691056 22346 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:46:32.690328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35701 (local address 127.0.0.1:45078)
0927 10:46:32.690510 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 10:46:32.690525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:46:32.690626 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:46:32.690936 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.195:35701: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 10:46:32.691846 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 21813
I20260927 10:46:32.698727 10627 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:46:33.044039 10627 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:46:33.044193 10627 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:46:33.389547 10627 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:46:33.389 UTC [21399] LOG: received smart shutdown request
2026-09-27 10:46:33.393 UTC [21399] LOG: background worker "logical replication launcher" (PID 21408) exited with exit code 1
2026-09-27 10:46:33.393 UTC [21403] LOG: shutting down
2026-09-27 10:46:33.572 UTC [21399] LOG: database system is shut down
2026-09-27T10:46:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45853 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:46:33 dist-test-slave-wmg0 krb5kdc[22385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:46:33 dist-test-slave-wmg0 krb5kdc[22385](info): set up 2 sockets
Sep 27 10:46:33 dist-test-slave-wmg0 krb5kdc[22385](info): commencing operation
krb5kdc: starting...
W20260927 10:46:34.741526 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:46:34 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790505994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:46:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/rangeradmin_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/rangeradmin_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/rangerlookup_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/rangerlookup_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
I20260927 10:46:34.912454 10627 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKx0Sgr/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/postgres -l logfile start
2026-09-27 10:46:37.469 UTC [22415] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 10:46:37.469 UTC [22415] LOG: listening on IPv4 address "127.10.96.212", port 56347
2026-09-27 10:46:37.473 UTC [22415] LOG: listening on Unix socket "/tmp/.s.PGSQL.56347"
2026-09-27 10:46:37.495 UTC [22418] LOG: database system was shut down at 2026-09-27 10:46:35 UTC
2026-09-27 10:46:37.498 UTC [22415] LOG: database system is ready to accept connections
I20260927 10:46:38.445269 10627 mini_postgres.cc:96] Postgres bound to 56347
2026-09-27 10:46:38.451 UTC [22426] FATAL: database "slave" does not exist
127.10.96.212:56347 - accepting connections
I20260927 10:46:38.452006 10627 mini_ranger.cc:161] Starting Ranger...
I20260927 10:46:38.466665 10627 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 10:46:38.980960 10627 mini_ranger.cc:87] Created ranger Postgres database
I20260927 10:46:38.981050 10627 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-admin
2026-09-27 10:46:39,250 [I] DB FLAVOR :POSTGRES
2026-09-27 10:46:39,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:46:39,251 [I] Checking connection..
2026-09-27 10:46:39,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 10:46:39,503 [I] Checking connection passed.
2026-09-27 10:46:39,504 [I] --------- Verifying version history table ---------
2026-09-27 10:46:39,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:46:39,738 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:46:39,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:46:39,995 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 10:46:39,995 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 10:46:39,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 10:46:40.235 UTC [22490] WARNING: there is no transaction in progress
2026-09-27 10:46:40,242 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 10:46:40,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 10:46:40,473 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 10:46:40,474 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:46:40,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 10:46:40,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 10:46:40,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wmg0.c.gcp-upstream.internal','N') ;"
2026-09-27 10:46:41,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:46:41,442 [I] Table x_portal_user does not exist in database ranger
2026-09-27 10:46:41,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:46:41,675 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 10:46:41,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:46:41,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:46:42,137 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 10:46:42,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 10:46:43.091 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.107 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.124 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.140 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.156 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.174 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.240 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.248 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.268 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.279 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.290 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.305 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.316 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.324 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.331 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:43.340 UTC [22598] WARNING: there is no transaction in progress
2026-09-27 10:46:44,124 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 10:46:44,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 10:46:44,398 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 10:46:44,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 10:46:44,676 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 10:46:44,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 10:46:44,922 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 10:46:44,923 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 10:46:45,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 10:46:45,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wmg0.c.gcp-upstream.internal';"
2026-09-27 10:46:45,694 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 10:46:45,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 10:46:45,958 [I] DB_PATCHES have already been applied
I20260927 10:46:45.963692 10627 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKx0Sgr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 10:46:45.963788 10627 mini_ranger.cc:193] Using host: 127.10.96.212
I20260927 10:46:45.967280 10627 mini_ranger.cc:227] Ranger admin URL: http://127.10.96.212:36555
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:46:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 10:46:47.074473 10627 mini_ranger.cc:160] Time spent starting Ranger: real 8.622s user 0.007s sys 0.003s
Sep 27, 2026 10:47:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:47:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:47:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:47:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:47:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 10:47:08.029875 10627 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/rangerkms_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/rangerkms_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.96.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.96.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.96.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/HTTP_127.10.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 10:47:08 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506028, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 10:47:08.114785 10627 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 10:47:08.124688 10627 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 10:47:08.968423 10627 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 10:47:08.968525 10627 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms
I20260927 10:47:08.968544 10627 mini_ranger_kms.cc:227] Using postgres at 127.10.96.212:56347
2026-09-27 10:47:09,143 [I] DB FLAVOR :POSTGRES
2026-09-27 10:47:09,143 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 10:47:09,144 [I] Checking connection
2026-09-27 10:47:09,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 10:47:09,407 [I] connection success
2026-09-27 10:47:09,407 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 10:47:09,407 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 10:47:09,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 10:47:09,654 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 10:47:09,655 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 10:47:09,655 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 10:47:09,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.96.212:56347/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 10:47:09,924 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 10:47:10.378278 10627 mini_ranger_kms.cc:310] Created kms service
I20260927 10:47:10.576661 10627 mini_ranger_kms.cc:326] Created kudu user
I20260927 10:47:10.631408 10627 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 10:47:10 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506028, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:11.079398 10627 mini_ranger_kms.cc:384] Added ranger policy
I20260927 10:47:11.079651 10627 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKx0Sgr/build/debug/bin/postgresql.jar:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKx0Sgr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 10:47:11.079679 10627 mini_ranger_kms.cc:240] Using host: 127.10.96.212
I20260927 10:47:11.084131 10627 mini_ranger_kms.cc:276] Ranger KMS PID: 22778
I20260927 10:47:11.084239 10627 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.96.212:41459
Sep 27, 2026 10:47:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:47:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:47:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:47:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:47:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:47:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 10:47:13 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506033, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.96.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:13.373658 10627 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.259s user 0.002s sys 0.158s
I20260927 10:47:13.373791 10627 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 10:47:13.373871 10627 mini_ranger_kms.cc:401] 127.10.96.212:41459/kms/v1/keys
Sep 27 10:47:13 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506033, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.96.212@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
Sep 27 10:47:14 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506033, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.96.212@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
Sep 27 10:47:16 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:47:16.352670 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:45161
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:45161
--ranger_config_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:16.467156 22829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:16.467496 22829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:16.467612 22829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:16.472167 22829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:16.472309 22829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:16.472376 22829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:16.472414 22829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:16.472456 22829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:16.479027 22829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.96.212:41459/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:45161
--ranger_config_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:16.480154 22829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:16.481277 22829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:16.490257 22837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:16.490299 22834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:16.490289 22835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:16.490458 22829 server_base.cc:1034] running on GCE node
I20260927 10:47:16.491238 22829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:16.492462 22829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:16.493642 22829 hybrid_clock.cc:648] HybridClock initialized: now 1790506036493626 us; error 38 us; skew 500 ppm
Sep 27 10:47:16 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506036, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:16.501907 22829 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:16.503050 22829 webserver.cc:460] Webserver started at http://127.10.96.254:41131/ using document root <none> and password file <none>
I20260927 10:47:16.503665 22829 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:16.503750 22829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:16.503983 22829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:47:16 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506036, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:16.553763 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "34b74b2d64a14f798c56e05f068c6fc2"
format_stamp: "Formatted at 2026-09-27 10:47:16 on dist-test-slave-wmg0"
server_key: "305f3c18e21c190ce380ea5fe215502a"
server_key_iv: "6673c4fd9ef774785f6e79a93a76cb58"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:16.554512 22829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "34b74b2d64a14f798c56e05f068c6fc2"
format_stamp: "Formatted at 2026-09-27 10:47:16 on dist-test-slave-wmg0"
server_key: "305f3c18e21c190ce380ea5fe215502a"
server_key_iv: "6673c4fd9ef774785f6e79a93a76cb58"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:16.559535 22829 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 10:47:16.574522 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:16.576180 22829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:16.576354 22829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "34b74b2d64a14f798c56e05f068c6fc2"
format_stamp: "Formatted at 2026-09-27 10:47:16 on dist-test-slave-wmg0"
server_key: "305f3c18e21c190ce380ea5fe215502a"
server_key_iv: "6673c4fd9ef774785f6e79a93a76cb58"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:16.576499 22829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:16.591714 22829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:16.595103 22829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:16.595311 22829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:16.604024 22829 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:45161
I20260927 10:47:16.604049 22896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:45161 every 8 connection(s)
I20260927 10:47:16.605346 22829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:16.608395 22897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:16.611186 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 22829
I20260927 10:47:16.611303 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:47:16.611621 10627 external_mini_cluster.cc:1428] Setting key 1a751632c8363326c9aac075c83f7a00
I20260927 10:47:16.614604 22897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Bootstrap starting.
I20260927 10:47:16.617146 22897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:16.617991 22897 log.cc:824] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:16.620093 22897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: No bootstrap required, opened a new log
Sep 27 10:47:16 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:16.624681 22897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:16.625036 22897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:16.625094 22897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b74b2d64a14f798c56e05f068c6fc2, State: Initialized, Role: FOLLOWER
I20260927 10:47:16.625684 22897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:16.625854 22897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:16.625954 22897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:16.626058 22897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:16.627313 22897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:16.627682 22897 leader_election.cc:302] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b74b2d64a14f798c56e05f068c6fc2; no voters:
I20260927 10:47:16.628062 22897 leader_election.cc:288] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:16.628183 22902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:16.628471 22902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 LEADER]: Becoming Leader. State: Replica: 34b74b2d64a14f798c56e05f068c6fc2, State: Running, Role: LEADER
I20260927 10:47:16.628756 22900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:16.613039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.10.96.254:45161)
0927 10:47:16.613536 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:16.613554 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:16.613597 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:47:16.614430 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:16.614454 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:16.614882 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:16.615253 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:16.616163 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:16.617824 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:16.618873 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:16.619596 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:16.619755 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:16.623046 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:16.623056 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:16.623074 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:16.623108 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:16.625289 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:16.625826 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:16.625832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:16.625838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:16.626016 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:16.626336 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:16.626340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:16.626342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:16.626928 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:16.627147 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:16.627384 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 10:47:16.627698 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
I20260927 10:47:16.629153 22902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:16.629410 22897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:16.630828 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b74b2d64a14f798c56e05f068c6fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } } }
I20260927 10:47:16.630955 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:16.631223 22904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b74b2d64a14f798c56e05f068c6fc2. Latest consensus state: current_term: 1 leader_uuid: "34b74b2d64a14f798c56e05f068c6fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } } }
I20260927 10:47:16.631336 22904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:16.631708 22897 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:47:16.631886 22914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:16.637907 22914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:16.644091 22914 catalog_manager.cc:1347] Generated new cluster ID: 872ac59fe9ba4913854721caf54e04ce
I20260927 10:47:16.644152 22914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:16.660626 22914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:16.662137 22914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:16.679153 22914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Generated new TSK 0
I20260927 10:47:16.680073 22914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:47:17.367070 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:45161
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:17.518539 22942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:17.518852 22942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:17.518945 22942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:17.523186 22942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:17.523280 22942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:17.523392 22942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:47:17.529250 22942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.96.212:41459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:17.530635 22942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:17.531781 22942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:17.543483 22950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:17.543499 22952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:17.544102 22949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:17.544597 22942 server_base.cc:1034] running on GCE node
I20260927 10:47:17.545120 22942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:17.545912 22942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:17.547091 22942 hybrid_clock.cc:648] HybridClock initialized: now 1790506037547062 us; error 45 us; skew 500 ppm
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:17.550239 22942 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:47:17.551249 22942 webserver.cc:460] Webserver started at http://127.10.96.193:39325/ using document root <none> and password file <none>
I20260927 10:47:17.551831 22942 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:17.551927 22942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:17.552174 22942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:17.568009 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "e5b0ec4ece1c5b98a1961c3ff6c42d33"
server_key_iv: "b721fddae02d53346b387b927b8992f0"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:17.568807 22942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "e5b0ec4ece1c5b98a1961c3ff6c42d33"
server_key_iv: "b721fddae02d53346b387b927b8992f0"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:17.574107 22942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 10:47:17.592924 22959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:17.594732 22942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:17.594867 22942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "e5b0ec4ece1c5b98a1961c3ff6c42d33"
server_key_iv: "b721fddae02d53346b387b927b8992f0"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:17.595001 22942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:17.623095 22942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:17.627050 22942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:17.627261 22942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:17.627871 22942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:17.628942 22942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:17.629042 22942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:17.629114 22942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:17.629146 22942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:17.640354 22942 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:34037
I20260927 10:47:17.640383 23074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:34037 every 8 connection(s)
I20260927 10:47:17.641319 22942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:17.650352 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 22942
I20260927 10:47:17.650498 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:17.650808 10627 external_mini_cluster.cc:1428] Setting key cf9ac664e43671b28bbc3615dcee0719
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:47:17.665936 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:17.645502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:39743 (local address 127.10.96.254:45161)
0927 10:47:17.647848 (+ 2346us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:17.647858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:17.647873 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:47:17.647919 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:17.647922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:17.647970 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:17.648060 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:17.649692 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.655033 (+ 5341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.656802 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.657236 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.658147 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:17.660745 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:17.660751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:17.660758 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:17.660795 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:17.662875 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:17.663637 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:17.663640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:17.663642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:17.663741 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:17.664203 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:17.664208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:17.664212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:17.664513 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:17.664618 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:17.665626 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0927 10:47:17.665756 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2264,"thread_start_us":92,"threads_started":1}
I20260927 10:47:17.666819 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:17.644314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45161 (local address 127.10.96.193:39743)
0927 10:47:17.646274 (+ 1960us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:17.646333 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:17.647394 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:17.648371 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:17.648380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:17.648753 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:17.649500 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:17.649522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.655223 (+ 5701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.655227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:17.656629 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:17.656643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.657441 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.657447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:17.657717 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:17.658509 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:17.658535 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:17.660505 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:17.663040 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:17.663049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:17.663071 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:17.663499 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:17.663852 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:17.663856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:17.663858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:17.664077 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:17.664677 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:17.664684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:17.665058 (+ 374us) client_negotiation.cc:769] Sending connection context
0927 10:47:17.665300 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 10:47:17.665749 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1716,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:47:17.668238 23075 heartbeater.cc:347] Connected to a master server at 127.10.96.254:45161
I20260927 10:47:17.668478 23075 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:17.669121 23075 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:17.671084 22861 ts_manager.cc:194] Registered new tserver with Master: 7ce3bc7e4e2f4e38a81aae9bda57f2a3 (127.10.96.193:34037)
I20260927 10:47:17.672260 22861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:39743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:47:17.686628 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:45161
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 10:47:17.816876 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:17.817245 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:17.817327 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:17.822705 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:17.822805 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:17.822911 23081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:47:17.828876 23081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.96.212:41459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:17.830170 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:17.831305 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:17.840970 23090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:17.841157 23088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:17.841645 23081 server_base.cc:1034] running on GCE node
W20260927 10:47:17.841579 23087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:17.842161 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:17.842844 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:17.844034 23081 hybrid_clock.cc:648] HybridClock initialized: now 1790506037844008 us; error 49 us; skew 500 ppm
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:17.847226 23081 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:47:17.848382 23081 webserver.cc:460] Webserver started at http://127.10.96.194:46269/ using document root <none> and password file <none>
I20260927 10:47:17.849108 23081 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:17.849192 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:17.849413 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:17.869144 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "09778512a8ca4bb8bb18e7a2efe83ef0"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "b73f011dc0d614ac290d8e21a55bdc77"
server_key_iv: "355eac1f9a8420254d8a8817ddef1ff5"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:17.869922 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "09778512a8ca4bb8bb18e7a2efe83ef0"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "b73f011dc0d614ac290d8e21a55bdc77"
server_key_iv: "355eac1f9a8420254d8a8817ddef1ff5"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:17.876607 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:17.906211 23097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:17.908144 23081 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260927 10:47:17.908334 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "09778512a8ca4bb8bb18e7a2efe83ef0"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "b73f011dc0d614ac290d8e21a55bdc77"
server_key_iv: "355eac1f9a8420254d8a8817ddef1ff5"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:17.908504 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:17.926271 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:17.930341 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:17.930543 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:17.931385 23081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:17.932444 23081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:17.932507 23081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:17.932572 23081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:17.932602 23081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:17.942994 23081 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:42535
I20260927 10:47:17.944293 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:17.949172 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23081
I20260927 10:47:17.949291 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:17.949641 10627 external_mini_cluster.cc:1428] Setting key 9d152b37eafc3e860327a40b8f71f65d
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:47:17.972205 23210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:42535 every 8 connection(s)
Sep 27 10:47:17 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:18.000736 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:17.968220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:41259 (local address 127.10.96.254:45161)
0927 10:47:17.968356 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:17.968361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:17.973124 (+ 4763us) server_negotiation.cc:407] Connection header received
0927 10:47:17.975682 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:17.975687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:17.975751 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:17.975818 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:17.978917 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.979801 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.987744 (+ 7943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.988140 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.988244 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:17.996136 (+ 7892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:17.996141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:17.996144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:17.996182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:17.997985 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:17.998725 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:17.998729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:17.998731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:17.998845 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:17.999293 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:17.999297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:17.999299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:17.999531 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:17.999609 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:18.000396 (+ 787us) server_negotiation.cc:299] Negotiation successful
0927 10:47:18.000565 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:47:18.001742 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:17.966961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45161 (local address 127.10.96.194:41259)
0927 10:47:17.972924 (+ 5963us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:17.972988 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:17.975367 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:17.977422 (+ 2055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:17.977434 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:17.977883 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:17.978667 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:17.978691 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.981542 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.981545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:17.986705 (+ 5160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:17.986723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:17.988986 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:17.988992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:17.989235 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:17.993967 (+ 4732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:17.993994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:17.995841 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:17.998143 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:17.998151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:17.998171 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:17.998550 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:17.998955 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:17.998958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:17.998960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:17.999167 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:17.999688 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:17.999696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:18.000048 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 10:47:18.000250 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 10:47:18.000614 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5685,"thread_start_us":178,"threads_started":1}
I20260927 10:47:18.003126 23211 heartbeater.cc:347] Connected to a master server at 127.10.96.254:45161
I20260927 10:47:18.003408 23211 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:18.004031 23211 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:47:18.005247 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:45161
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:18.005277 22861 ts_manager.cc:194] Registered new tserver with Master: 09778512a8ca4bb8bb18e7a2efe83ef0 (127.10.96.194:42535)
I20260927 10:47:18.006078 22861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:41259
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506037, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
W20260927 10:47:18.186365 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:18.186781 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:18.186862 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:18.201371 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:18.201578 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:18.201745 23216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:47:18.211025 23216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.96.212:41459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:18.212538 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:18.214018 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:18.228183 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:18.228708 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:18.229902 23216 server_base.cc:1034] running on GCE node
W20260927 10:47:18.229960 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:18.230639 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:18.231596 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:18.232779 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790506038232695 us; error 99 us; skew 500 ppm
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:18.237051 23216 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:47:18.238276 23216 webserver.cc:460] Webserver started at http://127.10.96.195:41031/ using document root <none> and password file <none>
I20260927 10:47:18.239105 23216 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:18.239202 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:18.239464 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:18.255733 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "98852b945ef846529c58b8fec30d175e"
format_stamp: "Formatted at 2026-09-27 10:47:18 on dist-test-slave-wmg0"
server_key: "c25bd9c5b99d097aad6cbafa2be898b6"
server_key_iv: "e7c5558171b7f5a8b09dcef2803e3b44"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:18.256524 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "98852b945ef846529c58b8fec30d175e"
format_stamp: "Formatted at 2026-09-27 10:47:18 on dist-test-slave-wmg0"
server_key: "c25bd9c5b99d097aad6cbafa2be898b6"
server_key_iv: "e7c5558171b7f5a8b09dcef2803e3b44"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:18.272131 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.006s sys 0.001s
I20260927 10:47:18.297996 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.300035 23216 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.004s sys 0.000s
I20260927 10:47:18.300208 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "98852b945ef846529c58b8fec30d175e"
format_stamp: "Formatted at 2026-09-27 10:47:18 on dist-test-slave-wmg0"
server_key: "c25bd9c5b99d097aad6cbafa2be898b6"
server_key_iv: "e7c5558171b7f5a8b09dcef2803e3b44"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:18.300382 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:18.318456 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:18.322516 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:18.322731 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:18.323513 23216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:18.324585 23216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:18.324647 23216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.324702 23216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:18.324728 23216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.337741 23216 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:37051
I20260927 10:47:18.337793 23344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:37051 every 8 connection(s)
I20260927 10:47:18.339593 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:18.344022 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23216
I20260927 10:47:18.344151 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:18.344457 10627 external_mini_cluster.cc:1428] Setting key e871f3ef93b72350874690d001c2b29c
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:18.360069 23078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:18.342171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:52291 (local address 127.10.96.254:45161)
0927 10:47:18.342314 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:18.342320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:18.343513 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 10:47:18.344886 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:18.344891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:18.344952 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:18.345010 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:18.347250 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.348193 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.350196 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.350600 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.350712 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:18.354105 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:18.354110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:18.354113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:18.354145 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:18.356230 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:18.356976 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:18.356980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:18.356982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:18.357085 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:18.357594 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:18.357598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:18.357601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:18.357867 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:18.358926 (+ 1059us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:18.359716 (+ 790us) server_negotiation.cc:299] Negotiation successful
0927 10:47:18.359880 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:47:18.361414 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:18.342745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45161 (local address 127.10.96.195:52291)
0927 10:47:18.343337 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:18.343395 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:18.344590 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:18.345501 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:18.345511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:18.345909 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:18.347075 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:18.347096 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.348360 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.348364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:18.350034 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:18.350047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.350795 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.350800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:18.351049 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:18.351855 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:18.351878 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:18.353861 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:18.356377 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:18.356384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:18.356407 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:18.356823 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:18.357214 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:18.357217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:18.357219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:18.357427 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:18.359010 (+ 1583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:18.359018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:18.359391 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 10:47:18.359628 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 10:47:18.360150 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":147,"threads_started":1}
I20260927 10:47:18.362710 23345 heartbeater.cc:347] Connected to a master server at 127.10.96.254:45161
I20260927 10:47:18.363051 23345 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:18.363722 23345 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:18.364943 22861 ts_manager.cc:194] Registered new tserver with Master: 98852b945ef846529c58b8fec30d175e (127.10.96.195:37051)
I20260927 10:47:18.365792 22861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:52291
I20260927 10:47:18.370524 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:18.371528 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 22942
I20260927 10:47:18.377274 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23081
I20260927 10:47:18.384104 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23216
I20260927 10:47:18.392385 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 22829
I20260927 10:47:18.400763 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:45161
--webserver_interface=127.10.96.254
--webserver_port=41131
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:45161
--ranger_config_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:18.511086 23348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:18.511363 23348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:18.511401 23348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:18.515369 23348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:18.515431 23348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:18.515450 23348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:18.515466 23348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:18.515482 23348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:18.520946 23348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:45161
--ranger_config_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:45161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:18.522063 23348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:18.523057 23348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:18.531605 23353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:18.531627 23354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:18.531589 23356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:18.531838 23348 server_base.cc:1034] running on GCE node
I20260927 10:47:18.532564 23348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:18.533787 23348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:18.534977 23348 hybrid_clock.cc:648] HybridClock initialized: now 1790506038534942 us; error 47 us; skew 500 ppm
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:18.538110 23348 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:18.539206 23348 webserver.cc:460] Webserver started at http://127.10.96.254:41131/ using document root <none> and password file <none>
I20260927 10:47:18.539915 23348 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:18.540015 23348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:18.598063 23348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 10:47:18.600848 23364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.602339 23348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:18.602460 23348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "34b74b2d64a14f798c56e05f068c6fc2"
format_stamp: "Formatted at 2026-09-27 10:47:16 on dist-test-slave-wmg0"
server_key: "305f3c18e21c190ce380ea5fe215502a"
server_key_iv: "6673c4fd9ef774785f6e79a93a76cb58"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:18.603178 23348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:18.619856 23348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:18.622701 23348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:18.622922 23348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:18.630412 23348 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:45161
I20260927 10:47:18.630483 23416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:45161 every 8 connection(s)
I20260927 10:47:18.631534 23348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:18.635998 23417 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:47:18.637714 23417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Bootstrap starting.
I20260927 10:47:18.638846 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23348
I20260927 10:47:18.639299 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:34037
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:45161
--webserver_port=39325
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:18.654790 23417 log.cc:824] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:18.661422 23417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:18.661897 23417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Bootstrap complete.
I20260927 10:47:18.666082 23417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:18.666456 23417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b74b2d64a14f798c56e05f068c6fc2, State: Initialized, Role: FOLLOWER
I20260927 10:47:18.666961 23417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:18.667075 23417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:18.667160 23417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:18.667248 23417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:47:18.669169 23417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:18.669555 23417 leader_election.cc:302] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b74b2d64a14f798c56e05f068c6fc2; no voters:
I20260927 10:47:18.669922 23417 leader_election.cc:288] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:47:18.670014 23421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:47:18.670217 23421 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [term 2 LEADER]: Becoming Leader. State: Replica: 34b74b2d64a14f798c56e05f068c6fc2, State: Running, Role: LEADER
I20260927 10:47:18.670516 23421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } }
I20260927 10:47:18.671149 23417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:18.671968 23423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b74b2d64a14f798c56e05f068c6fc2. Latest consensus state: current_term: 2 leader_uuid: "34b74b2d64a14f798c56e05f068c6fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } } }
I20260927 10:47:18.672094 23423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:18.671972 23422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34b74b2d64a14f798c56e05f068c6fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b74b2d64a14f798c56e05f068c6fc2" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 45161 } } }
I20260927 10:47:18.672286 23422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:18.672689 23429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:18.675870 23429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:18.676364 23417 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 10:47:18.676673 23429 catalog_manager.cc:1259] Loaded cluster ID: 872ac59fe9ba4913854721caf54e04ce
I20260927 10:47:18.676792 23429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:18.679207 23429 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:18.679840 23429 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Loaded TSK: 0
I20260927 10:47:18.681005 23429 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:47:18.782869 23419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:18.783236 23419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:18.783300 23419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:18.789881 23419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:18.789963 23419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:18.790050 23419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:47:18.798174 23419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:34037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=39325
--tserver_master_addrs=127.10.96.254:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:18.799611 23419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:18.800961 23419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:18.813764 23454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:18.814110 23419 server_base.cc:1034] running on GCE node
W20260927 10:47:18.814348 23457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:18.814450 23455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:18.814980 23419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:18.815850 23419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:18.817037 23419 hybrid_clock.cc:648] HybridClock initialized: now 1790506038817009 us; error 43 us; skew 500 ppm
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:18.820715 23419 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:47:18.821924 23419 webserver.cc:460] Webserver started at http://127.10.96.193:39325/ using document root <none> and password file <none>
I20260927 10:47:18.822646 23419 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:18.822750 23419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:18.847898 23419 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260927 10:47:18.851372 23465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.853211 23419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:18.853358 23419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "e5b0ec4ece1c5b98a1961c3ff6c42d33"
server_key_iv: "b721fddae02d53346b387b927b8992f0"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:18.854192 23419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:18.889114 23419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:18.893481 23419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:18.893707 23419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:18.894487 23419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:18.895591 23419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:18.895656 23419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.895712 23419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:18.895742 23419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:18.907097 23419 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:34037
I20260927 10:47:18.908551 23419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:18.909346 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:34037 every 8 connection(s)
I20260927 10:47:18.918421 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23419
I20260927 10:47:18.918944 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:42535
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:45161
--webserver_port=46269
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:47:18 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506038, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:18.941409 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:18.915959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45161 (local address 127.10.96.193:58761)
0927 10:47:18.917197 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:18.917254 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:18.918429 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:18.920455 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:18.920467 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:18.920841 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:18.921385 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:18.921399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.929126 (+ 7727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.929131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:18.930779 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:18.930795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.932123 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.932127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:18.932338 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:18.933111 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:18.933133 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:18.934983 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:18.937696 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:18.937702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:18.937719 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:18.938081 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:18.938493 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:18.938499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:18.938500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:18.938701 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:18.939529 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:18.939536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:18.939843 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 10:47:18.940033 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 10:47:18.940368 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":158,"threads_started":1}
I20260927 10:47:18.943625 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:18.915741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:58761 (local address 127.10.96.254:45161)
0927 10:47:18.916235 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:18.916258 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:18.917419 (+ 1161us) server_negotiation.cc:407] Connection header received
0927 10:47:18.919078 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:18.919094 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:18.919547 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:18.919928 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:18.921626 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.923299 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.930960 (+ 7661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:18.931910 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:18.932121 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:18.935246 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:18.935254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:18.935276 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:18.935311 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:18.937545 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:18.938227 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:18.938233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:18.938239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:18.938375 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:18.938858 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:18.938863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:18.938865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:18.939383 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:18.942219 (+ 2836us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:18.942386 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 10:47:18.942703 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":137,"threads_started":1}
I20260927 10:47:18.946719 23579 heartbeater.cc:347] Connected to a master server at 127.10.96.254:45161
I20260927 10:47:18.947005 23579 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:18.947644 23579 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
W20260927 10:47:18.949959 23579 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:45161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:47:18.950093 23579 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:18.950286 23579 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:18.951122 23579 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:18.951297 23579 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
W20260927 10:47:18.952380 23579 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:47:19.065817 23583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:19.066123 23583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:19.066201 23583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:19.070603 23583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:19.070694 23583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:19.070787 23583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:47:19.076622 23583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:42535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=46269
--tserver_master_addrs=127.10.96.254:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:19.077929 23583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:19.079072 23583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:19.090138 23589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:19.090238 23583 server_base.cc:1034] running on GCE node
W20260927 10:47:19.090137 23591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:19.090333 23588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:19.091008 23583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:19.091818 23583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:19.093029 23583 hybrid_clock.cc:648] HybridClock initialized: now 1790506039092994 us; error 41 us; skew 500 ppm
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:19.096515 23583 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:47:19.097779 23583 webserver.cc:460] Webserver started at http://127.10.96.194:46269/ using document root <none> and password file <none>
I20260927 10:47:19.098532 23583 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:19.098639 23583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:19.122895 23583 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 10:47:19.129895 23599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:19.131667 23583 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260927 10:47:19.131814 23583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "09778512a8ca4bb8bb18e7a2efe83ef0"
format_stamp: "Formatted at 2026-09-27 10:47:17 on dist-test-slave-wmg0"
server_key: "b73f011dc0d614ac290d8e21a55bdc77"
server_key_iv: "355eac1f9a8420254d8a8817ddef1ff5"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:19.132556 23583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:19.151484 23583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:19.154856 23583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:19.155067 23583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:19.155689 23583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:19.156608 23583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:19.156673 23583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:19.156744 23583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:19.156781 23583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:19.169478 23583 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:42535
I20260927 10:47:19.169555 23712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:42535 every 8 connection(s)
I20260927 10:47:19.170920 23583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:19.179019 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23583
I20260927 10:47:19.179549 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.96.212:41459/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:37051
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:45161
--webserver_port=41031
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:46825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:19.196239 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:19.175214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:35903 (local address 127.10.96.254:45161)
0927 10:47:19.175408 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:19.175413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:19.182077 (+ 6664us) server_negotiation.cc:407] Connection header received
0927 10:47:19.183474 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:19.183478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:19.183548 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:19.183658 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:19.185373 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.186204 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.187976 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.188346 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.188428 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:19.191641 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:19.191645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:19.191648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:19.191679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:19.193580 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:19.194228 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:19.194232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:19.194234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:19.194353 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:19.194821 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:19.194825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:19.194827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:19.195067 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:19.195149 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:19.195902 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 10:47:19.196059 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 10:47:19.198226 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:19.174120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45161 (local address 127.10.96.194:35903)
0927 10:47:19.181851 (+ 7731us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:19.181925 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:19.183192 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:19.184015 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:19.184027 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:19.184417 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:19.185139 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:19.185161 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.186390 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.186395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:19.187825 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:19.187836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.188645 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.188649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:19.188851 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:19.189606 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:19.189629 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:19.191418 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:19.193714 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:19.193721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:19.193738 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:19.194103 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:19.194503 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:19.194506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:19.194509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:19.194700 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:19.195235 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:19.195242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:19.195595 (+ 353us) client_negotiation.cc:769] Sending connection context
0927 10:47:19.196685 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0927 10:47:19.197073 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7411,"thread_start_us":137,"threads_started":1}
I20260927 10:47:19.199816 23713 heartbeater.cc:347] Connected to a master server at 127.10.96.254:45161
I20260927 10:47:19.200054 23713 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.200623 23713 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
W20260927 10:47:19.202455 23713 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:45161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:47:19.202582 23713 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.202755 23713 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:19.203661 23713 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.203816 23713 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
W20260927 10:47:19.204609 23713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:47:19.335098 23716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:19.335362 23716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:19.335438 23716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:19.340185 23716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:19.340262 23716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:19.340361 23716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:47:19.347424 23716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.96.212:41459/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:37051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=41031
--tserver_master_addrs=127.10.96.254:45161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:19.348886 23716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:19.350472 23716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:19.362993 23723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:19.363123 23722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:19.363049 23725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:19.364511 23716 server_base.cc:1034] running on GCE node
I20260927 10:47:19.365113 23716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:19.365960 23716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:19.367149 23716 hybrid_clock.cc:648] HybridClock initialized: now 1790506039367110 us; error 59 us; skew 500 ppm
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:19.370874 23716 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:47:19.372097 23716 webserver.cc:460] Webserver started at http://127.10.96.195:41031/ using document root <none> and password file <none>
I20260927 10:47:19.372900 23716 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:19.373070 23716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:19.390216 23716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 10:47:19.393071 23733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:19.394788 23716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:47:19.395026 23716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "98852b945ef846529c58b8fec30d175e"
format_stamp: "Formatted at 2026-09-27 10:47:18 on dist-test-slave-wmg0"
server_key: "c25bd9c5b99d097aad6cbafa2be898b6"
server_key_iv: "e7c5558171b7f5a8b09dcef2803e3b44"
server_key_version: "kuduclusterkey@0"
I20260927 10:47:19.395684 23716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:19.415035 23716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:19.418802 23716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:19.419077 23716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:19.419863 23716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:19.420980 23716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:19.421044 23716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:19.421097 23716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:19.421167 23716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:19.433169 23716 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:37051
I20260927 10:47:19.433645 23846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:37051 every 8 connection(s)
I20260927 10:47:19.434574 23716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:19.439380 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23716
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:19.454624 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:19.436547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:50537 (local address 127.10.96.254:45161)
0927 10:47:19.436692 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:19.436697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:19.437945 (+ 1248us) server_negotiation.cc:407] Connection header received
0927 10:47:19.439292 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:19.439297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:19.439358 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:19.439413 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:19.441513 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.442335 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.443614 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.443848 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.443979 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:19.450184 (+ 6205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:19.450189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:19.450191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:19.450225 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:19.452088 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:19.452696 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:19.452703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:19.452708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:19.452810 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:19.453335 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:19.453338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:19.453340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:19.453613 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:19.453717 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:19.454331 (+ 614us) server_negotiation.cc:299] Negotiation successful
0927 10:47:19.454461 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:47:19.455773 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:19.437037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:45161 (local address 127.10.96.195:50537)
0927 10:47:19.437763 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:19.437819 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:19.439022 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:19.439881 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:19.439891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:19.440609 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:19.441306 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:19.441325 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.442505 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.442509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:19.443480 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:19.443490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.444024 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.444028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:19.444209 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:19.444862 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:19.444877 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:19.449926 (+ 5049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:19.452238 (+ 2312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:19.452244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:19.452269 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:19.452568 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:19.452942 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:19.452948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:19.452950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:19.453201 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:19.453794 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:19.453800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:19.454073 (+ 273us) client_negotiation.cc:769] Sending connection context
0927 10:47:19.454289 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 10:47:19.454639 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":142,"threads_started":1}
I20260927 10:47:19.457746 23847 heartbeater.cc:347] Connected to a master server at 127.10.96.254:45161
I20260927 10:47:19.458014 23847 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.458616 23847 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
W20260927 10:47:19.460623 23847 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:45161 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 10:47:19.460754 23847 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.460937 23847 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:19.461812 23847 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.461988 23847 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:19.462522 23851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:19.440120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.10.96.254:45161)
0927 10:47:19.450485 (+ 10365us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:19.450496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:19.450510 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:47:19.450550 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:19.450553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:19.450603 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:19.450723 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:19.453564 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.454383 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.455782 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:19.456181 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:19.456291 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:19.458187 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:19.458193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:19.458197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:19.458221 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:19.460170 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:19.460705 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:19.460710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:19.460711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:19.460796 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:19.461092 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:19.461095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:19.461097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:19.461333 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:19.461565 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:19.462227 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 10:47:19.462364 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10260,"thread_start_us":99,"threads_started":1}
W20260927 10:47:19.462831 23847 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 10:47:19 dist-test-slave-wmg0 krb5kdc[22385](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:19.952654 23579 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:19.952980 23579 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:19.954596 23381 ts_manager.cc:194] Registered new tserver with Master: 7ce3bc7e4e2f4e38a81aae9bda57f2a3 (127.10.96.193:34037)
I20260927 10:47:19.956281 23381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:58761
I20260927 10:47:20.204838 23713 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:20.205149 23713 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:20.206014 23381 ts_manager.cc:194] Registered new tserver with Master: 09778512a8ca4bb8bb18e7a2efe83ef0 (127.10.96.194:42535)
I20260927 10:47:20.206801 23381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:35903
Sep 27 10:47:20 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
Sep 27 10:47:20 dist-test-slave-wmg0 krb5kdc[22385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506039, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for HTTP/127.10.96.212@KRBTEST.COM
I20260927 10:47:20.463073 23847 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:20.463359 23847 heartbeater.cc:511] Master 127.10.96.254:45161 requested a full tablet report, sending...
I20260927 10:47:20.464111 23381 ts_manager.cc:194] Registered new tserver with Master: 98852b945ef846529c58b8fec30d175e (127.10.96.195:37051)
I20260927 10:47:20.464895 23381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:50537
I20260927 10:47:20.471012 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:20.486929 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.476154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.10.96.254:45161)
0927 10:47:20.476564 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.476567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.477078 (+ 511us) server_negotiation.cc:407] Connection header received
0927 10:47:20.477133 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.477136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.477199 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.477288 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:20.478139 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.478944 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.480279 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.480615 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.480737 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.482489 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:20.482493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:20.482496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:20.482521 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:20.484338 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:20.484829 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:20.484834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:20.484836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:20.484924 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:20.485443 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:20.485446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:20.485448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:20.485734 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:20.485882 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.486591 (+ 709us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.486721 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":94,"threads_started":1}
I20260927 10:47:20.491671 23381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:20.496140 23381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:20.529211 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.518640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.10.96.193:34037)
0927 10:47:20.523723 (+ 5083us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.523731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.524714 (+ 983us) server_negotiation.cc:407] Connection header received
0927 10:47:20.525002 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.525008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.525182 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.525362 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:20.525860 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.526636 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.527893 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.528454 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.528577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.528734 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.528964 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.529066 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4961,"thread_start_us":88,"threads_started":1}
I20260927 10:47:20.530036 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.517831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34037 (local address 127.0.0.1:50652)
0927 10:47:20.523986 (+ 6155us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.524599 (+ 613us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:20.524773 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:20.525561 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:20.525569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:20.525593 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:20.525729 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.525737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.526752 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.526758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.527759 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.527768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.528577 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.528580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.528684 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.528719 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:47:20.528836 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 10:47:20.528963 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5970,"thread_start_us":100,"threads_started":1}
I20260927 10:47:20.532917 23511 tablet_service.cc:1467] Processing CreateTablet for tablet 62b6da62237c46eebe048b7f00593c80 (DEFAULT_TABLE table=test-table [id=ab89ed76ca1a421ba2f902013fe24218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:20.534063 23511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b6da62237c46eebe048b7f00593c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:20.534895 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.517931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:42535 (local address 127.0.0.1:59398)
0927 10:47:20.526878 (+ 8947us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.526893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:20.526969 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:20.530435 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:20.530438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:20.530455 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:20.530541 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.530547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.531930 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.531934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.533130 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.533142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.534472 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.534476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.534570 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.534593 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:47:20.534666 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 10:47:20.534734 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8881,"thread_start_us":77,"threads_started":1}
I20260927 10:47:20.535595 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.518113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.10.96.194:42535)
0927 10:47:20.529728 (+ 11615us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.529735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.529768 (+ 33us) server_negotiation.cc:407] Connection header received
0927 10:47:20.529867 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.529873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.530105 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.530273 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:20.530666 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.531774 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.533279 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.534151 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.534259 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.534406 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.535357 (+ 951us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.535459 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11488,"thread_start_us":125,"threads_started":1}
I20260927 10:47:20.538759 23647 tablet_service.cc:1467] Processing CreateTablet for tablet 62b6da62237c46eebe048b7f00593c80 (DEFAULT_TABLE table=test-table [id=ab89ed76ca1a421ba2f902013fe24218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:20.539938 23647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b6da62237c46eebe048b7f00593c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:20.553575 23884 tablet_bootstrap.cc:492] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: Bootstrap starting.
I20260927 10:47:20.556095 23884 tablet_bootstrap.cc:654] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:20.556218 23883 tablet_bootstrap.cc:492] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Bootstrap starting.
I20260927 10:47:20.557088 23884 log.cc:824] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:20.559460 23884 tablet_bootstrap.cc:492] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: No bootstrap required, opened a new log
I20260927 10:47:20.559669 23884 ts_tablet_manager.cc:1397] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260927 10:47:20.564127 23884 raft_consensus.cc:348] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.564419 23884 raft_consensus.cc:374] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:20.564471 23884 raft_consensus.cc:729] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce3bc7e4e2f4e38a81aae9bda57f2a3, State: Initialized, Role: FOLLOWER
I20260927 10:47:20.565244 23884 consensus_queue.cc:260] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.566022 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.522376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.10.96.195:37051)
0927 10:47:20.533776 (+ 11400us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.533783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.533808 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:47:20.534714 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.534721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.534922 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.535070 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:20.561552 (+ 26482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.562805 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.564311 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.565137 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.565220 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.565372 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.565758 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.565878 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11285,"thread_start_us":113,"threads_started":1}
I20260927 10:47:20.566548 23884 ts_tablet_manager.cc:1428] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260927 10:47:20.566620 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.521975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:37051 (local address 127.0.0.1:39572)
0927 10:47:20.530728 (+ 8753us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.530744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:20.530824 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:20.561172 (+ 30348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:20.561176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:20.561209 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:20.561327 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.561335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.562977 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.562981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.564154 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.564167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.565497 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.565502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.565589 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.565622 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 10:47:20.566394 (+ 772us) client_negotiation.cc:240] Negotiation successful
0927 10:47:20.566487 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8695,"thread_start_us":89,"threads_started":1}
I20260927 10:47:20.566991 23579 heartbeater.cc:503] Master 127.10.96.254:45161 was elected leader, sending a full tablet report...
I20260927 10:47:20.570763 23781 tablet_service.cc:1467] Processing CreateTablet for tablet 62b6da62237c46eebe048b7f00593c80 (DEFAULT_TABLE table=test-table [id=ab89ed76ca1a421ba2f902013fe24218]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:20.572109 23781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b6da62237c46eebe048b7f00593c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:20.577117 23883 tablet_bootstrap.cc:654] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:20.578123 23883 log.cc:824] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:20.581619 23887 tablet_bootstrap.cc:492] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Bootstrap starting.
I20260927 10:47:20.583941 23887 tablet_bootstrap.cc:654] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:20.585335 23887 log.cc:824] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:20.587841 23883 tablet_bootstrap.cc:492] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: No bootstrap required, opened a new log
I20260927 10:47:20.587939 23887 tablet_bootstrap.cc:492] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: No bootstrap required, opened a new log
I20260927 10:47:20.588120 23883 ts_tablet_manager.cc:1397] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.004s
I20260927 10:47:20.588183 23887 ts_tablet_manager.cc:1397] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:47:20.592353 23887 raft_consensus.cc:348] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.592648 23887 raft_consensus.cc:374] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:20.592700 23887 raft_consensus.cc:729] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98852b945ef846529c58b8fec30d175e, State: Initialized, Role: FOLLOWER
I20260927 10:47:20.593223 23887 consensus_queue.cc:260] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.593451 23883 raft_consensus.cc:348] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.593791 23883 raft_consensus.cc:374] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:20.593892 23883 raft_consensus.cc:729] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09778512a8ca4bb8bb18e7a2efe83ef0, State: Initialized, Role: FOLLOWER
I20260927 10:47:20.594374 23883 consensus_queue.cc:260] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.595112 23887 ts_tablet_manager.cc:1428] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260927 10:47:20.595419 23883 ts_tablet_manager.cc:1428] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:20.595497 23847 heartbeater.cc:503] Master 127.10.96.254:45161 was elected leader, sending a full tablet report...
I20260927 10:47:20.595778 23713 heartbeater.cc:503] Master 127.10.96.254:45161 was elected leader, sending a full tablet report...
W20260927 10:47:20.665109 23580 tablet.cc:2369] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:47:20.683348 23714 tablet.cc:2369] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:47:20.689805 23848 tablet.cc:2369] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:20.857522 23890 raft_consensus.cc:482] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:20.857753 23890 raft_consensus.cc:504] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.858992 23890 leader_election.cc:288] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce3bc7e4e2f4e38a81aae9bda57f2a3 (127.10.96.193:34037), 09778512a8ca4bb8bb18e7a2efe83ef0 (127.10.96.194:42535)
I20260927 10:47:20.863842 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.859393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:35337 (local address 127.10.96.193:34037)
0927 10:47:20.859520 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.859525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.859690 (+ 165us) server_negotiation.cc:407] Connection header received
0927 10:47:20.859957 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.859966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.860037 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.860108 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:20.860430 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.861195 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.862823 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.863208 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.863354 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.863411 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.863605 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.863695 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:47:20.863919 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.859288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:42535 (local address 127.10.96.195:36737)
0927 10:47:20.859627 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.859644 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:20.859749 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:20.860048 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:20.860051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:20.860071 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:20.860243 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.860249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.861155 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.861162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.862650 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.862660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.863500 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.863506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.863613 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.863631 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:47:20.863698 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 10:47:20.863764 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":99,"spinlock_wait_cycles":384,"thread_start_us":113,"threads_started":1}
I20260927 10:47:20.863934 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.859674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:36737 (local address 127.10.96.194:42535)
0927 10:47:20.859798 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.859803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.859835 (+ 32us) server_negotiation.cc:407] Connection header received
0927 10:47:20.859891 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.859894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.859940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.860023 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:20.860384 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.861049 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.862800 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.863398 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.863516 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.863564 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.863730 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.863809 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:47:20.864487 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.859208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34037 (local address 127.10.96.195:35337)
0927 10:47:20.859599 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.859613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:20.859749 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:20.860154 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:20.860157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:20.860175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:20.860298 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.860304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.861304 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.861307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.862649 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:20.862660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.863336 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.863340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:20.863445 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.863476 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:47:20.864263 (+ 787us) client_negotiation.cc:240] Negotiation successful
0927 10:47:20.864339 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":37,"spinlock_wait_cycles":5376,"thread_start_us":141,"threads_started":1}
I20260927 10:47:20.864748 23667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b6da62237c46eebe048b7f00593c80" candidate_uuid: "98852b945ef846529c58b8fec30d175e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" is_pre_election: true
I20260927 10:47:20.865091 23667 raft_consensus.cc:2454] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98852b945ef846529c58b8fec30d175e in term 0.
I20260927 10:47:20.865622 23734 leader_election.cc:302] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09778512a8ca4bb8bb18e7a2efe83ef0, 98852b945ef846529c58b8fec30d175e; no voters:
I20260927 10:47:20.865916 23890 raft_consensus.cc:2781] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:20.865935 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b6da62237c46eebe048b7f00593c80" candidate_uuid: "98852b945ef846529c58b8fec30d175e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" is_pre_election: true
I20260927 10:47:20.866024 23890 raft_consensus.cc:482] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:20.866070 23890 raft_consensus.cc:3037] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:20.866204 23533 raft_consensus.cc:2454] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98852b945ef846529c58b8fec30d175e in term 0.
I20260927 10:47:20.867321 23890 raft_consensus.cc:504] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.867714 23890 leader_election.cc:288] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [CANDIDATE]: Term 1 election: Requested vote from peers 7ce3bc7e4e2f4e38a81aae9bda57f2a3 (127.10.96.193:34037), 09778512a8ca4bb8bb18e7a2efe83ef0 (127.10.96.194:42535)
I20260927 10:47:20.868047 23533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b6da62237c46eebe048b7f00593c80" candidate_uuid: "98852b945ef846529c58b8fec30d175e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3"
I20260927 10:47:20.868105 23667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b6da62237c46eebe048b7f00593c80" candidate_uuid: "98852b945ef846529c58b8fec30d175e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0"
I20260927 10:47:20.868166 23533 raft_consensus.cc:3037] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:20.868209 23667 raft_consensus.cc:3037] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:20.869422 23667 raft_consensus.cc:2454] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98852b945ef846529c58b8fec30d175e in term 1.
I20260927 10:47:20.869422 23533 raft_consensus.cc:2454] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98852b945ef846529c58b8fec30d175e in term 1.
I20260927 10:47:20.869727 23735 leader_election.cc:302] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce3bc7e4e2f4e38a81aae9bda57f2a3, 98852b945ef846529c58b8fec30d175e; no voters:
I20260927 10:47:20.869910 23890 raft_consensus.cc:2781] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:20.870164 23890 raft_consensus.cc:686] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 1 LEADER]: Becoming Leader. State: Replica: 98852b945ef846529c58b8fec30d175e, State: Running, Role: LEADER
I20260927 10:47:20.870523 23890 consensus_queue.cc:237] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } }
I20260927 10:47:20.874231 23381 catalog_manager.cc:5495] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e reported cstate change: term changed from 0 to 1, leader changed from <none> to 98852b945ef846529c58b8fec30d175e (127.10.96.195). New cstate: current_term: 1 leader_uuid: "98852b945ef846529c58b8fec30d175e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98852b945ef846529c58b8fec30d175e" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 37051 } health_report { overall_health: HEALTHY } } }
I20260927 10:47:20.886933 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.882927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.10.96.195:37051)
0927 10:47:20.883063 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:20.883067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:20.883082 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:47:20.883157 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:20.883160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:20.883220 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:20.883330 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:20.883575 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.884369 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.885204 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:20.885545 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:20.885664 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:20.886030 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:20.886560 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:20.886680 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:20.886716 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 10:47:20.886770 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:47:20.893570 23533 raft_consensus.cc:1261] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3 [term 1 FOLLOWER]: Refusing update from remote peer 98852b945ef846529c58b8fec30d175e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:20.893570 23667 raft_consensus.cc:1261] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 1 FOLLOWER]: Refusing update from remote peer 98852b945ef846529c58b8fec30d175e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:20.894374 23890 consensus_queue.cc:1035] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [LEADER]: Connected to new peer: Peer: permanent_uuid: "09778512a8ca4bb8bb18e7a2efe83ef0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:20.894603 23901 consensus_queue.cc:1035] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce3bc7e4e2f4e38a81aae9bda57f2a3" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:20.902578 23904 mvcc.cc:205] Tried to move back new op lower bound from 7333912743492001792 to 7333912743408492544. Current Snapshot: MvccSnapshot[applied={T|T < 7333912743492001792}]
I20260927 10:47:20.904580 23905 mvcc.cc:205] Tried to move back new op lower bound from 7333912743492001792 to 7333912743408492544. Current Snapshot: MvccSnapshot[applied={T|T < 7333912743492001792}]
I20260927 10:47:20.906741 23906 mvcc.cc:205] Tried to move back new op lower bound from 7333912743492001792 to 7333912743408492544. Current Snapshot: MvccSnapshot[applied={T|T < 7333912743492001792}]
I20260927 10:47:20.913817 23381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:20.914005 23381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:20.916487 23381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b74b2d64a14f798c56e05f068c6fc2: Sending DeleteTablet for 3 replicas of tablet 62b6da62237c46eebe048b7f00593c80
I20260927 10:47:20.917303 23511 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b6da62237c46eebe048b7f00593c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:20 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:50652
I20260927 10:47:20.917855 23647 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b6da62237c46eebe048b7f00593c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:20 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:59398
I20260927 10:47:20.917842 23781 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b6da62237c46eebe048b7f00593c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:20 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:39572
I20260927 10:47:20.918906 23914 tablet_replica.cc:330] T 62b6da62237c46eebe048b7f00593c80 P 7ce3bc7e4e2f4e38a81aae9bda57f2a3: stopping tablet replica
I20260927 10:47:20.919166 23915 tablet_replica.cc:330] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: stopping tablet replica
I20260927 10:47:20.919298 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23419
I20260927 10:47:20.919593 23915 raft_consensus.cc:2229] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:20.919939 23915 raft_consensus.cc:2258] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:20.925961 23915 ts_tablet_manager.cc:1905] T 62b6da62237c46eebe048b7f00593c80 P 09778512a8ca4bb8bb18e7a2efe83ef0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 10:47:20.927837 23366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:34037 (error 108)
I20260927 10:47:20.928685 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23583
I20260927 10:47:20.931174 23916 tablet_replica.cc:330] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: stopping tablet replica
I20260927 10:47:20.931528 23916 raft_consensus.cc:2229] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:20.932003 23916 raft_consensus.cc:2258] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:20.933730 23916 ts_tablet_manager.cc:1905] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:20.937381 23916 ts_tablet_manager.cc:1918] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:20.937573 23916 log.cc:1196] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/62b6da62237c46eebe048b7f00593c80
I20260927 10:47:20.938014 23916 ts_tablet_manager.cc:1939] T 62b6da62237c46eebe048b7f00593c80 P 98852b945ef846529c58b8fec30d175e: Deleting consensus metadata
I20260927 10:47:20.938190 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23716
I20260927 10:47:20.939643 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.939247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34037 (local address 127.0.0.1:50660)
0927 10:47:20.939463 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.939549 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 10:47:20.939782 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:20.939247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:42535 (local address 127.0.0.1:59406)
0927 10:47:20.939672 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:20.939719 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:42535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":261}
W20260927 10:47:20.939872 23366 catalog_manager.cc:4584] TS 7ce3bc7e4e2f4e38a81aae9bda57f2a3 (127.10.96.193:34037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62b6da62237c46eebe048b7f00593c80: Network error: Client connection negotiation failed: client connection to 127.10.96.193:34037: connect: Connection refused (error 111)
I20260927 10:47:20.945111 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23348
I20260927 10:47:20.952559 10627 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 10:47:21.297655 10627 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 10:47:21.297765 10627 mini_ranger.cc:66] Stopping Ranger...
I20260927 10:47:21.642673 10627 mini_ranger.cc:68] Stopped Ranger
2026-09-27 10:47:21.642 UTC [22415] LOG: received smart shutdown request
2026-09-27 10:47:21.646 UTC [22415] LOG: background worker "logical replication launcher" (PID 22424) exited with exit code 1
2026-09-27 10:47:21.646 UTC [22419] LOG: shutting down
2026-09-27 10:47:21.804 UTC [22415] LOG: database system is shut down
2026-09-27T10:47:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48240 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:21 dist-test-slave-wmg0 krb5kdc[23926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:21 dist-test-slave-wmg0 krb5kdc[23926](info): set up 2 sockets
Sep 27 10:47:21 dist-test-slave-wmg0 krb5kdc[23926](info): commencing operation
krb5kdc: starting...
W20260927 10:47:22.981817 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:47:23.100602 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33451
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33451,127.10.96.253:46115,127.10.96.252:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:23.211448 23939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:23.211776 23939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:23.211863 23939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:23.216780 23939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:23.216918 23939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:23.216961 23939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:23.217000 23939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:23.217073 23939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:23.222522 23939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33451,127.10.96.253:46115,127.10.96.252:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:23.223681 23939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:23.224754 23939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:23.233891 23944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:23.233942 23945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:23.233891 23947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:23.234112 23939 server_base.cc:1034] running on GCE node
I20260927 10:47:23.234855 23939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:23.236068 23939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:23.237264 23939 hybrid_clock.cc:648] HybridClock initialized: now 1790506043237223 us; error 58 us; skew 500 ppm
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:23.240190 23939 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:23.241104 23939 webserver.cc:460] Webserver started at http://127.10.96.254:44999/ using document root <none> and password file <none>
I20260927 10:47:23.241750 23939 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:23.241843 23939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:23.242074 23939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:23.244118 23939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "921f874f352e40b9b73ac45d471e8561"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "af3e03dd0d277a42d889dcc232410516"
server_key_iv: "609596459bd3c95760e2ef41b6b8744d"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.244679 23939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "921f874f352e40b9b73ac45d471e8561"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "af3e03dd0d277a42d889dcc232410516"
server_key_iv: "609596459bd3c95760e2ef41b6b8744d"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.248508 23939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:23.251065 23954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:23.252409 23939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:23.252565 23939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "921f874f352e40b9b73ac45d471e8561"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "af3e03dd0d277a42d889dcc232410516"
server_key_iv: "609596459bd3c95760e2ef41b6b8744d"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.252684 23939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:23.277710 23939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:23.281000 23939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:23.281208 23939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:23.288648 23939 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:33451
I20260927 10:47:23.288666 24006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:33451 every 8 connection(s)
I20260927 10:47:23.289685 23939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:23.292596 24007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:23.296440 24007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.296996 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 23939
I20260927 10:47:23.297139 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:47:23.297407 10627 external_mini_cluster.cc:1428] Setting key 851429f7270d5068f2a3f6e8186b2f3c
WARNING: no policy specified for kudu/127.10.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.253@KRBTEST.COM" created.
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:23.314201 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.299277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.10.96.254:33451)
0927 10:47:23.299723 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.299746 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.300029 (+ 283us) server_negotiation.cc:407] Connection header received
0927 10:47:23.301056 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.301060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.301238 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.301437 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.302401 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.303271 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.304250 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.304619 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.304793 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.308558 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.308568 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.308585 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.308617 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.310438 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.311093 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.311104 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.311105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.311216 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.311805 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.311810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.311812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.312268 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.312388 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.312947 (+ 559us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.313257 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":291,"thread_start_us":158,"threads_started":1}
I20260927 10:47:23.314204 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.299278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33451 (local address 127.0.0.1:47052)
0927 10:47:23.299878 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.299919 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.300750 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.301560 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.301565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.301996 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.302251 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.302267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.303415 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.303418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.304130 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.304139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.304773 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.304777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.304882 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.305548 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.305563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.308391 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.310562 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.310575 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.310588 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.310961 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.311326 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.311332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.311337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.311506 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.312409 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.312419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.312622 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.312867 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.313385 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":113,"spinlock_wait_cycles":20224,"thread_start_us":161,"threads_started":1}
I20260927 10:47:23.316959 24007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.317843 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.317536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:45820)
0927 10:47:23.317702 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.317779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 10:47:23.318045 23957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
I20260927 10:47:23.319029 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.318778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:45832)
0927 10:47:23.318913 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.318976 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 10:47:23.319175 24007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:46115: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab.
Entry for principal kudu/127.10.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab.
I20260927 10:47:23.325644 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:46115
--webserver_interface=127.10.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab
--principal=kudu/127.10.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33451,127.10.96.253:46115,127.10.96.252:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:23.368441 24007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } attempt: 1
I20260927 10:47:23.369388 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.369038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:45836)
0927 10:47:23.369213 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.369288 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:47:23.370187 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.369961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:45840)
0927 10:47:23.370068 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.370120 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:47:23.370324 24007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:46115: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
I20260927 10:47:23.438500 24007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } attempt: 2
I20260927 10:47:23.439368 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.439049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:59216)
0927 10:47:23.439221 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.439289 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:47:23.440152 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.439963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:59230)
0927 10:47:23.440058 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.440103 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:47:23.440348 24007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:46115: Network error: Client connection negotiation failed: client connection to 127.10.96.253:46115: connect: Connection refused (error 111)
W20260927 10:47:23.441322 24014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:23.441658 24014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:23.441710 24014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:23.445966 24014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:23.446022 24014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:23.446041 24014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:23.446059 24014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:23.446112 24014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:23.451762 24014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33451,127.10.96.253:46115,127.10.96.252:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab
--principal=kudu/127.10.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:46115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:23.452895 24014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:23.454032 24014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:23.462839 24022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:23.462842 24020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:23.462884 24014 server_base.cc:1034] running on GCE node
W20260927 10:47:23.462842 24019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:23.463732 24014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:23.464814 24014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:23.466001 24014 hybrid_clock.cc:648] HybridClock initialized: now 1790506043465986 us; error 33 us; skew 500 ppm
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:23.469049 24014 init.cc:377] Logged in from keytab as kudu/127.10.96.253@KRBTEST.COM (short username kudu)
I20260927 10:47:23.470124 24014 webserver.cc:460] Webserver started at http://127.10.96.253:40257/ using document root <none> and password file <none>
I20260927 10:47:23.470726 24014 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:23.470834 24014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:23.471071 24014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:23.473299 24014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data/instance:
uuid: "91c130c87274484e968735b9a2b668fc"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "d0044a232152ab59c79460f517c18f42"
server_key_iv: "12023fe78a0b0d229be975582b082714"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.473986 24014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal/instance:
uuid: "91c130c87274484e968735b9a2b668fc"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "d0044a232152ab59c79460f517c18f42"
server_key_iv: "12023fe78a0b0d229be975582b082714"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.477978 24014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:47:23.480782 24029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:23.482142 24014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:23.482297 24014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal
uuid: "91c130c87274484e968735b9a2b668fc"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "d0044a232152ab59c79460f517c18f42"
server_key_iv: "12023fe78a0b0d229be975582b082714"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.482421 24014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:23.491972 24014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:23.495440 24014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:23.495651 24014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:23.502929 24014 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.253:46115
I20260927 10:47:23.502954 24081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.253:46115 every 8 connection(s)
I20260927 10:47:23.503907 24014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
I20260927 10:47:23.506634 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:23.510492 24082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.512151 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24014
I20260927 10:47:23.512315 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-1/wal/instance
I20260927 10:47:23.512634 10627 external_mini_cluster.cc:1428] Setting key fa2e60090b788173edbe4adf3deba568
WARNING: no policy specified for kudu/127.10.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.252@KRBTEST.COM" created.
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:23.527532 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.512489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.10.96.254:33451)
0927 10:47:23.512670 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.512675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.513686 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 10:47:23.514651 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.514656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.514720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.514844 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.516078 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.516618 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.517855 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.518108 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.518224 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.523503 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.523506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.523509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.523534 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.525335 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.525911 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.525915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.525917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.526004 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.526357 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.526361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.526363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.526562 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.526665 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.527212 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.527353 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:47:23.528342 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.512965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33451 (local address 127.0.0.1:59322)
0927 10:47:23.513515 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.513570 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.514436 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.515038 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.515046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.515409 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.515899 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.515913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.516771 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.516774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.517728 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.517737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.518264 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.518268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.518453 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.519086 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.519102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.523307 (+ 4205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.525455 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.525481 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.525501 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.525797 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.526099 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.526104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.526106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.526252 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.526676 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.526684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.526956 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.527144 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.527486 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":149,"threads_started":1}
I20260927 10:47:23.530011 24082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.531507 24007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } attempt: 3
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab.
Entry for principal kudu/127.10.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab.
I20260927 10:47:23.540638 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:34881
--webserver_interface=127.10.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab
--principal=kudu/127.10.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33451,127.10.96.253:46115,127.10.96.252:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:23.542305 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.531965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:59248)
0927 10:47:23.532206 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.532220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.532337 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.532973 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.532976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.533218 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.533356 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.533366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.534458 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.534461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.535766 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.535776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.536351 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.536355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.536478 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.536959 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.536975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.538323 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.540485 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.540488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.540489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.540678 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.541035 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.541037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.541039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.541118 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.541718 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.541721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.541815 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.541962 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.542116 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 10:47:23.542500 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.532178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.10.96.253:46115)
0927 10:47:23.532681 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.532706 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.532720 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:47:23.532768 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.532773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.532824 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.532927 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.533499 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.534335 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.535859 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.536196 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.536338 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.538478 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.538482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.538484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.538508 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.540370 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.540835 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.540844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.540848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.540937 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.541208 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.541214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.541219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.541581 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.541722 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.542095 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.542312 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":153,"threads_started":1}
I20260927 10:47:23.544592 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.530611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:59232)
0927 10:47:23.530732 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.530752 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.530841 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.531614 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.531620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.531795 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.531956 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.531963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.533444 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.533448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.534324 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.534334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.535191 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.535194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.535321 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.535854 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.535868 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.537451 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.543029 (+ 5578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.543033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.543036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.543274 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.543558 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.543561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.543563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.543656 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.544103 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.544106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.544196 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.544308 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.544445 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":42}
I20260927 10:47:23.544723 24007 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.544842 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.530786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.10.96.253:46115)
0927 10:47:23.531220 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.531233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.531270 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:47:23.531306 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.531311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.531463 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.531594 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.532077 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.533293 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.534458 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.535072 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.535194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.538154 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.538162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.538176 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.538196 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.542907 (+ 4711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.543393 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.543398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.543403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.543480 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.543760 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.543766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.543771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.544001 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.544102 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.544389 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.544520 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":297,"thread_start_us":164,"threads_started":1}
I20260927 10:47:23.545591 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.545264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37324)
0927 10:47:23.545415 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.545507 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 10:47:23.545614 24082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.546321 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.546053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37330)
0927 10:47:23.546203 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.546272 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 10:47:23.546418 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.546053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37342)
0927 10:47:23.546241 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.546314 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 10:47:23.546516 24033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
W20260927 10:47:23.546622 24007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34881: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
I20260927 10:47:23.547391 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.547187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37352)
0927 10:47:23.547305 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.547347 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 10:47:23.547551 24082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34881: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
I20260927 10:47:23.577781 24007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } attempt: 1
I20260927 10:47:23.578671 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.578379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37356)
0927 10:47:23.578532 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.578596 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 10:47:23.579416 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.579203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37358)
0927 10:47:23.579310 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.579364 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:47:23.579543 24007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34881: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
I20260927 10:47:23.596802 24082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } attempt: 1
I20260927 10:47:23.597748 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.597406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37360)
0927 10:47:23.597610 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.597674 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260927 10:47:23.598519 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.598304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37368)
0927 10:47:23.598414 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.598466 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:47:23.598644 24082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34881: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
I20260927 10:47:23.654740 24007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } attempt: 2
I20260927 10:47:23.655597 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.655298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37384)
0927 10:47:23.655448 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.655523 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 10:47:23.656383 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.656164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37398)
0927 10:47:23.656280 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.656331 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:47:23.656530 24007 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34881: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
W20260927 10:47:23.657413 24089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:23.657706 24089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:23.657786 24089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:23.662094 24089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:23.662150 24089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:23.662171 24089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:23.662189 24089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:23.662232 24089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:23.666827 24082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } attempt: 2
I20260927 10:47:23.667631 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.667348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37404)
0927 10:47:23.667494 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.667561 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 10:47:23.667682 24089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33451,127.10.96.253:46115,127.10.96.252:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab
--principal=kudu/127.10.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:23.668432 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.668228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37412)
0927 10:47:23.668335 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.668381 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:47:23.668668 24082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34881: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34881: connect: Connection refused (error 111)
I20260927 10:47:23.669029 24089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:23.670135 24089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:23.678452 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:23.678603 24095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:23.678705 24096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:23.678946 24089 server_base.cc:1034] running on GCE node
I20260927 10:47:23.679564 24089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:23.680658 24089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:23.681823 24089 hybrid_clock.cc:648] HybridClock initialized: now 1790506043681789 us; error 48 us; skew 500 ppm
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:23.684553 24089 init.cc:377] Logged in from keytab as kudu/127.10.96.252@KRBTEST.COM (short username kudu)
I20260927 10:47:23.685495 24089 webserver.cc:460] Webserver started at http://127.10.96.252:40119/ using document root <none> and password file <none>
I20260927 10:47:23.686105 24089 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:23.686167 24089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:23.686342 24089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:23.688421 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data/instance:
uuid: "eed0c61e6fef40288e6760f3b2f027e2"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "86321e0b305b69254b077828ac771b70"
server_key_iv: "06f0fb1d409a72ba7e23570acf7102d7"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.688911 24089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal/instance:
uuid: "eed0c61e6fef40288e6760f3b2f027e2"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "86321e0b305b69254b077828ac771b70"
server_key_iv: "06f0fb1d409a72ba7e23570acf7102d7"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.692533 24089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:47:23.695086 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:23.696369 24089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:47:23.696527 24089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal
uuid: "eed0c61e6fef40288e6760f3b2f027e2"
format_stamp: "Formatted at 2026-09-27 10:47:23 on dist-test-slave-wmg0"
server_key: "86321e0b305b69254b077828ac771b70"
server_key_iv: "06f0fb1d409a72ba7e23570acf7102d7"
server_key_version: "encryptionkey@0"
I20260927 10:47:23.696646 24089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:23.713677 24089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:23.716979 24089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:23.717211 24089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:23.724689 24089 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.252:34881
I20260927 10:47:23.724700 24157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.252:34881 every 8 connection(s)
I20260927 10:47:23.725742 24089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
I20260927 10:47:23.728066 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24089
I20260927 10:47:23.728211 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790505884435945-10627-0/minicluster-data/master-2/wal/instance
I20260927 10:47:23.728519 10627 external_mini_cluster.cc:1428] Setting key ac1834211a71430f612d5202865d315a
I20260927 10:47:23.728765 24158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:23.733762 24158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:23.743975 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.729906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.10.96.252:34881)
0927 10:47:23.730344 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.730361 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.730398 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:47:23.731075 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.731091 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.731433 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.731723 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.732654 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.733823 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.734961 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.735651 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.735841 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.738833 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.738839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.738850 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.738879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.740969 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.741508 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.741521 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.741525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.741659 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.742013 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.742016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.742017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.742383 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.742620 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.742917 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.743288 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":156,"threads_started":1}
I20260927 10:47:23.745781 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.735746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.10.96.254:33451)
0927 10:47:23.735923 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.735930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.736319 (+ 389us) server_negotiation.cc:407] Connection header received
0927 10:47:23.736407 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.736410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.736463 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.736559 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.737314 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.737969 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.739074 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.739508 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.739597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.742190 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.742193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.742196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.742220 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.743844 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.744415 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.744421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.744423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.744510 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.744816 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.744819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.744821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.745018 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.745131 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.745512 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.745629 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 10:47:23.746059 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.735681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33451 (local address 127.0.0.1:59326)
0927 10:47:23.736133 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.736189 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.736319 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.736642 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.736656 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.737042 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.737170 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.737176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.738105 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.738109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.738942 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.738951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.739628 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.739634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.739803 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.740202 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.740221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.742040 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.743980 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.743991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.743997 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.744305 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.744617 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.744620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.744622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.744711 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.745143 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.745152 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.745266 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.745415 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.745851 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":154,"threads_started":1}
I20260927 10:47:23.747361 24158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:47:23.755716 24007 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } attempt: 3
I20260927 10:47:23.756265 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.746046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.10.96.254:33451)
0927 10:47:23.746197 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.746203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.746241 (+ 38us) server_negotiation.cc:407] Connection header received
0927 10:47:23.746385 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.746388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.746442 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.746512 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.747352 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.748047 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.749111 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.749450 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.749598 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.752499 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.752503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.752506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.752530 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.754457 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.754916 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.754921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.754923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.755010 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.755311 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.755315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.755317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.755509 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.755745 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.755893 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.756081 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":99,"threads_started":1}
I20260927 10:47:23.756589 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.747997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.10.96.253:46115)
0927 10:47:23.748160 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.748165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.748178 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:23.748221 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.748223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.748263 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.748323 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.748797 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.749512 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.750703 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.751015 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.751105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.753307 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.753310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.753312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.753335 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.754775 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.755231 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.755235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.755237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.755316 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.755597 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.755600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.755601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.755829 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.755956 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.756308 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.756437 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:47:23.757579 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.747876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:46115 (local address 127.0.0.1:59260)
0927 10:47:23.748017 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.748035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.748112 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.748380 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.748383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.748558 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.748670 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.748677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.749623 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.749627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.750531 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.750545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.751153 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.751161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.751287 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.751735 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.751752 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.753159 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.754936 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.754939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.754941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.755127 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.755416 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.755419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.755421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.755499 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.756001 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.756004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.756117 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.757216 (+ 1099us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.757379 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 10:47:23.758270 24158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:47:23.759871 24082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } attempt: 3
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:47:23.765913 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.756123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37428)
0927 10:47:23.756258 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.756273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.756337 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.756782 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.756785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.756953 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.757050 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.757056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.758192 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.758195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.758890 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.758899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.759535 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.759539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.759625 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.760175 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.760190 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.761644 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.763760 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.763763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.763765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.764013 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.764344 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.764349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.764352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.764459 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.765324 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.765328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.765439 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.765603 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.765756 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:47:23 dist-test-slave-wmg0 krb5kdc[23926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
I20260927 10:47:23.766510 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.756385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.10.96.252:34881)
0927 10:47:23.756492 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.756497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.756508 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:47:23.756538 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.756541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.756576 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.756648 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.757171 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.758051 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.759019 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.759409 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.759548 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.761780 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.761784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.761787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.761809 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.763642 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.764115 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.764121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.764124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.764216 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.764590 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.764594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.764596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.765205 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.765337 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.766267 (+ 930us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.766362 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":36}
I20260927 10:47:23.770263 24007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: Bootstrap starting.
I20260927 10:47:23.771335 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.758755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37434)
0927 10:47:23.759192 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.759207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.759275 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.759830 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.759833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.759980 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.760104 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.760114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.763066 (+ 2952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.763069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.763868 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.763876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.764608 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.764611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.764715 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.765205 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.765219 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.766752 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.769060 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.769066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.769071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.769574 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.770188 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.770191 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.770193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.770332 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.770792 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.770795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.770892 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.770987 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.771116 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":5,"spinlock_wait_cycles":4736}
I20260927 10:47:23.771536 24084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.760233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34881 (local address 127.0.0.1:37450)
0927 10:47:23.760493 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:23.760508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:23.760587 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:23.760851 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:23.760854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:23.761054 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:23.761175 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.761185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.762371 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.762376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.763177 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:23.763187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.763677 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.763681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:23.763767 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.764408 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:23.764423 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:23.767119 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:23.769750 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.769755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.769757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.770048 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.770453 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:23.770456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:23.770458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:23.770545 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:23.771025 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:23.771028 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:23.771131 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 10:47:23.771218 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:47:23.771345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 10:47:23.771867 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.758964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.10.96.252:34881)
0927 10:47:23.759255 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.759263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.759276 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:23.759366 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.759369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.759414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.759497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.762116 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.762943 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.763996 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.764397 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.764504 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.766907 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.766913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.766917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.766945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.768942 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.769695 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.769701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.769703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.769788 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.770440 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.770443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.770445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.770668 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.770757 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.771535 (+ 778us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.771654 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":222,"thread_start_us":104,"threads_started":1}
I20260927 10:47:23.772583 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.757974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59266 (local address 127.10.96.253:46115)
0927 10:47:23.758084 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.758088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.758156 (+ 68us) server_negotiation.cc:407] Connection header received
0927 10:47:23.758308 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.758310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.758348 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.758444 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.759300 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.760029 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.761142 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.761486 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.761570 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.766986 (+ 5416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.766992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.766997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.767021 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.770687 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.771166 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.771169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.771171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.771251 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.771560 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.771563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.771565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.771774 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.771993 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.772349 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.772453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:47:23.773293 24007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:23.774086 24007 log.cc:824] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:23.774833 24158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: Bootstrap starting.
I20260927 10:47:23.776371 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:23.776507 24007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: No bootstrap required, opened a new log
I20260927 10:47:23.777661 24158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:23.778625 24158 log.cc:824] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:23.779189 24165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.760271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.10.96.252:34881)
0927 10:47:23.760576 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.760586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.760600 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:47:23.760681 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.760685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.760723 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.760836 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.761379 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.762215 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.763313 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.763557 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.763678 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.767395 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.767399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.767403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.767434 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.769612 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.770188 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.770191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.770193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.770332 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.770666 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.770669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.770672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.770902 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.771989 (+ 1087us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.772064 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.772175 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":239,"spinlock_wait_cycles":5760,"thread_start_us":102,"threads_started":1}
I20260927 10:47:23.780303 24007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.780575 24007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:23.780646 24007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 921f874f352e40b9b73ac45d471e8561, State: Initialized, Role: FOLLOWER
I20260927 10:47:23.781165 24007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.782469 24082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: Bootstrap starting.
I20260927 10:47:23.782539 24007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 10:47:23.783499 23960 tablet.cc:2369] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:23.786675 24082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: Neither blocks nor log segments found. Creating new log.
W20260927 10:47:23.789041 24186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:23.789265 24186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:23.790506 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.779578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.10.96.253:46115)
0927 10:47:23.779679 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.779683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.780483 (+ 800us) server_negotiation.cc:407] Connection header received
0927 10:47:23.780659 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.780663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.780714 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.780764 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.781694 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.782375 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.783514 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.783811 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.783902 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.785976 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.785979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.785982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.786007 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.788395 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.788962 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.788965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.788967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.789067 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.789394 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.789399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.789403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.789657 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.789754 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.790204 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.790334 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:47:23.790518 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.779551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.10.96.254:33451)
0927 10:47:23.779669 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.779676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.779691 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:47:23.779728 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.779731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.779781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.779865 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.780909 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.781767 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.782997 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.783347 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.783437 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.785815 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.785819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.785822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.785853 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.788324 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.788955 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.788959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.788961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.789049 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.789378 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.789381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.789383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.789652 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.789760 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.790214 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.790352 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:47:23.791044 24175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.791281 24175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:47:23.788743 23939 master_runner.cc:186] Error getting master registration for 127.10.96.253:46115: OK, instance_id { permanent_uuid: "91c130c87274484e968735b9a2b668fc" instance_seqno: 1790506043500749 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:47:23.792598 23939 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:47:23.793728 24082 log.cc:824] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:23.795063 24158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: No bootstrap required, opened a new log
I20260927 10:47:23.796039 24082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: No bootstrap required, opened a new log
I20260927 10:47:23.799741 24158 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.800113 24158 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:23.800184 24158 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed0c61e6fef40288e6760f3b2f027e2, State: Initialized, Role: FOLLOWER
I20260927 10:47:23.800712 24158 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.801115 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:23.779916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.10.96.252:34881)
0927 10:47:23.780011 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:23.780014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:23.791151 (+ 11137us) server_negotiation.cc:407] Connection header received
0927 10:47:23.791339 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:23.791345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:23.791405 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:23.791467 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:23.792473 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.793338 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.794543 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:23.794946 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:23.795034 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:23.796973 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:23.796978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:23.796980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:23.797007 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:23.798969 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.799513 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.799517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.799519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.799632 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:23.800001 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:23.800008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:23.800012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:23.800247 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:23.800397 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:23.800818 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 10:47:23.800955 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:47:23.801661 24082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.801968 24082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:23.802069 24082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c130c87274484e968735b9a2b668fc, State: Initialized, Role: FOLLOWER
I20260927 10:47:23.802527 24158 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:23.802589 24082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.803181 24190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.803355 24190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:47:23.804208 24082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:23.805639 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.805820 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 10:47:23.809056 24201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:23.809177 24201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:23.811913 24089 master_runner.cc:263] No leader master found from master eed0c61e6fef40288e6760f3b2f027e2
I20260927 10:47:23.811990 24089 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:47:23.814136 24014 master_runner.cc:263] No leader master found from master 91c130c87274484e968735b9a2b668fc
I20260927 10:47:23.814194 24014 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:47:23.815412 24213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:23.815500 24213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:23.840178 24175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:23.840378 24175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.842197 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "921f874f352e40b9b73ac45d471e8561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c130c87274484e968735b9a2b668fc" is_pre_election: true
I20260927 10:47:23.842525 24056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 921f874f352e40b9b73ac45d471e8561 in term 0.
I20260927 10:47:23.843672 23957 leader_election.cc:302] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c130c87274484e968735b9a2b668fc, 921f874f352e40b9b73ac45d471e8561; no voters:
I20260927 10:47:23.845067 24175 leader_election.cc:288] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91c130c87274484e968735b9a2b668fc (127.10.96.253:46115), eed0c61e6fef40288e6760f3b2f027e2 (127.10.96.252:34881)
I20260927 10:47:23.845175 24175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:23.845250 24175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:23.845289 24175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:23.846606 24175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.846978 24175 leader_election.cc:288] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [CANDIDATE]: Term 1 election: Requested vote from peers 91c130c87274484e968735b9a2b668fc (127.10.96.253:46115), eed0c61e6fef40288e6760f3b2f027e2 (127.10.96.252:34881)
I20260927 10:47:23.847792 24131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "921f874f352e40b9b73ac45d471e8561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed0c61e6fef40288e6760f3b2f027e2"
I20260927 10:47:23.848071 24131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:23.848145 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "921f874f352e40b9b73ac45d471e8561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91c130c87274484e968735b9a2b668fc"
I20260927 10:47:23.848270 24056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:23.849504 24131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 921f874f352e40b9b73ac45d471e8561 in term 1.
I20260927 10:47:23.849716 24056 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 921f874f352e40b9b73ac45d471e8561 in term 1.
I20260927 10:47:23.849977 23958 leader_election.cc:302] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 921f874f352e40b9b73ac45d471e8561, eed0c61e6fef40288e6760f3b2f027e2; no voters:
I20260927 10:47:23.847796 24132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "921f874f352e40b9b73ac45d471e8561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed0c61e6fef40288e6760f3b2f027e2" is_pre_election: true
I20260927 10:47:23.850528 24132 raft_consensus.cc:2362] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 921f874f352e40b9b73ac45d471e8561 in term 1. Re-sending same reply.
I20260927 10:47:23.850713 24215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:23.850827 24215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [term 1 LEADER]: Becoming Leader. State: Replica: 921f874f352e40b9b73ac45d471e8561, State: Running, Role: LEADER
I20260927 10:47:23.851238 24215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } }
I20260927 10:47:23.853507 24175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 921f874f352e40b9b73ac45d471e8561. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.853674 24175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:23.854121 24217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:23.855077 24217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:23.864780 24132 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [term 1 FOLLOWER]: Refusing update from remote peer 921f874f352e40b9b73ac45d471e8561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:23.865567 24215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:23.869823 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 921f874f352e40b9b73ac45d471e8561. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.869969 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:47:23.871250 24056 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [term 1 FOLLOWER]: Refusing update from remote peer 921f874f352e40b9b73ac45d471e8561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:23.874423 24215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:23.878913 24190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 921f874f352e40b9b73ac45d471e8561. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.879045 24190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:47:23.884795 24215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.884951 24215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:23.886399 24190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.886518 24190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:47:23.889693 24218 mvcc.cc:205] Tried to move back new op lower bound from 7333912755662942208 to 7333912755618197504. Current Snapshot: MvccSnapshot[applied={T|T < 7333912755662942208}]
I20260927 10:47:23.889945 24221 mvcc.cc:205] Tried to move back new op lower bound from 7333912755662942208 to 7333912755618197504. Current Snapshot: MvccSnapshot[applied={T|T < 7333912755662942208}]
I20260927 10:47:23.892867 24175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.893007 24175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:23.894253 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "921f874f352e40b9b73ac45d471e8561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "921f874f352e40b9b73ac45d471e8561" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33451 } } peers { permanent_uuid: "91c130c87274484e968735b9a2b668fc" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 46115 } } peers { permanent_uuid: "eed0c61e6fef40288e6760f3b2f027e2" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34881 } } }
I20260927 10:47:23.894373 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:47:23.895151 24219 mvcc.cc:205] Tried to move back new op lower bound from 7333912755662942208 to 7333912755618197504. Current Snapshot: MvccSnapshot[applied={T|T < 7333912755662942208}]
I20260927 10:47:23.900398 24217 catalog_manager.cc:1347] Generated new cluster ID: b1d2c61ec9e946fca8fe89908f2e0538
I20260927 10:47:23.900465 24217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:23.942370 24217 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:23.943828 24217 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:23.973186 24217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 921f874f352e40b9b73ac45d471e8561: Generated new TSK 0
I20260927 10:47:23.974227 24217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:24.810523 24201 catalog_manager.cc:1259] Loaded cluster ID: b1d2c61ec9e946fca8fe89908f2e0538
I20260927 10:47:24.810612 24201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: loading cluster ID for follower catalog manager: success
I20260927 10:47:24.812215 24201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: acquiring CA information for follower catalog manager: success
I20260927 10:47:24.813308 24201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eed0c61e6fef40288e6760f3b2f027e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:47:24.816617 24213 catalog_manager.cc:1259] Loaded cluster ID: b1d2c61ec9e946fca8fe89908f2e0538
I20260927 10:47:24.816715 24213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: loading cluster ID for follower catalog manager: success
I20260927 10:47:24.818280 24213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: acquiring CA information for follower catalog manager: success
I20260927 10:47:24.819365 24213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91c130c87274484e968735b9a2b668fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:47:25.044155 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 23939
I20260927 10:47:25.050761 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24014
I20260927 10:47:25.056849 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24089
2026-09-27T10:47:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3126 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:25 dist-test-slave-wmg0 krb5kdc[24231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:25 dist-test-slave-wmg0 krb5kdc[24231](info): set up 2 sockets
Sep 27 10:47:25 dist-test-slave-wmg0 krb5kdc[24231](info): commencing operation
krb5kdc: starting...
W20260927 10:47:26.093737 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:47:26 dist-test-slave-wmg0 krb5kdc[24231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:47:26.214238 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:40465
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:26.324769 24244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:26.325047 24244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:26.325088 24244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:26.329159 24244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:26.329227 24244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:26.329247 24244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:26.329264 24244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:26.329280 24244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:26.334619 24244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:26.335726 24244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:26.336807 24244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:26.345141 24249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:26.345135 24250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:26.345319 24244 server_base.cc:1034] running on GCE node
W20260927 10:47:26.345498 24252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:26.346122 24244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:26.347177 24244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:26.348342 24244 hybrid_clock.cc:648] HybridClock initialized: now 1790506046348311 us; error 44 us; skew 500 ppm
Sep 27 10:47:26 dist-test-slave-wmg0 krb5kdc[24231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506046, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:26.351157 24244 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:26.352075 24244 webserver.cc:460] Webserver started at http://127.10.96.254:46781/ using document root <none> and password file <none>
I20260927 10:47:26.352645 24244 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:26.352746 24244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:26.352957 24244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:26.355130 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "97fdacdd19dd40be9c82857a232b4757"
format_stamp: "Formatted at 2026-09-27 10:47:26 on dist-test-slave-wmg0"
server_key: "8bb5eb58d0a70987f90024d73960b9ea"
server_key_iv: "febb84543f4e48cf0974c12a1a7fdbcb"
server_key_version: "encryptionkey@0"
I20260927 10:47:26.355742 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "97fdacdd19dd40be9c82857a232b4757"
format_stamp: "Formatted at 2026-09-27 10:47:26 on dist-test-slave-wmg0"
server_key: "8bb5eb58d0a70987f90024d73960b9ea"
server_key_iv: "febb84543f4e48cf0974c12a1a7fdbcb"
server_key_version: "encryptionkey@0"
I20260927 10:47:26.359623 24244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:26.362280 24259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:26.363626 24244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:47:26.363821 24244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "97fdacdd19dd40be9c82857a232b4757"
format_stamp: "Formatted at 2026-09-27 10:47:26 on dist-test-slave-wmg0"
server_key: "8bb5eb58d0a70987f90024d73960b9ea"
server_key_iv: "febb84543f4e48cf0974c12a1a7fdbcb"
server_key_version: "encryptionkey@0"
I20260927 10:47:26.363952 24244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:26.380117 24244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:26.383651 24244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:26.383874 24244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:26.391250 24244 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:40465
I20260927 10:47:26.391276 24311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:40465 every 8 connection(s)
I20260927 10:47:26.392254 24244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:26.395169 24312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:26.400516 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24244
I20260927 10:47:26.400619 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:47:26.400923 10627 external_mini_cluster.cc:1428] Setting key a19fc172fa8d23add32a0efd134a93c0
I20260927 10:47:26.400975 24312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757: Bootstrap starting.
I20260927 10:47:26.403604 24312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:26.404341 24312 log.cc:824] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:26.406625 24312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757: No bootstrap required, opened a new log
Sep 27 10:47:26 dist-test-slave-wmg0 krb5kdc[24231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:26.410917 24312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fdacdd19dd40be9c82857a232b4757" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40465 } }
I20260927 10:47:26.411134 24312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:26.411191 24312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fdacdd19dd40be9c82857a232b4757, State: Initialized, Role: FOLLOWER
I20260927 10:47:26.411624 24312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fdacdd19dd40be9c82857a232b4757" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40465 } }
I20260927 10:47:26.411796 24312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:26.411866 24312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:26.411984 24312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:26.413081 24312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fdacdd19dd40be9c82857a232b4757" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40465 } }
I20260927 10:47:26.413431 24312 leader_election.cc:302] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97fdacdd19dd40be9c82857a232b4757; no voters:
I20260927 10:47:26.413793 24312 leader_election.cc:288] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:26.413945 24317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:26.414163 24317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [term 1 LEADER]: Becoming Leader. State: Replica: 97fdacdd19dd40be9c82857a232b4757, State: Running, Role: LEADER
I20260927 10:47:26.414539 24317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fdacdd19dd40be9c82857a232b4757" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40465 } }
I20260927 10:47:26.415002 24312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:26.415788 24317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97fdacdd19dd40be9c82857a232b4757. Latest consensus state: current_term: 1 leader_uuid: "97fdacdd19dd40be9c82857a232b4757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fdacdd19dd40be9c82857a232b4757" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40465 } } }
I20260927 10:47:26.415987 24317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:26.416224 24318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97fdacdd19dd40be9c82857a232b4757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fdacdd19dd40be9c82857a232b4757" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 40465 } } }
I20260927 10:47:26.416325 24318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757 [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:26.416388 24326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:26.417160 24315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:26.402291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.10.96.254:40465)
0927 10:47:26.402741 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:26.402758 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:26.402794 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:47:26.403435 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:26.403453 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:26.403841 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:26.404172 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:26.405043 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:26.406489 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:26.407887 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:26.408368 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:26.408545 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:26.411405 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:26.411431 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:26.411450 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:26.411485 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:26.413303 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:26.413835 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:26.413842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:26.413850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:26.413998 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:26.414364 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:26.414366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:26.414369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:26.414867 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:26.415707 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:26.415859 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 10:47:26.416174 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":129,"threads_started":1}
I20260927 10:47:26.419559 24326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:26.420368 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:47:26.425170 24326 catalog_manager.cc:1347] Generated new cluster ID: 42ef1351b69c4bf081125d295c5f23a6
I20260927 10:47:26.425244 24326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:26.433193 24326 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:26.434150 24326 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:26.439014 24326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97fdacdd19dd40be9c82857a232b4757: Generated new TSK 0
I20260927 10:47:26.439661 24326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:47:26.476761 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24244
2026-09-27T10:47:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:26 dist-test-slave-wmg0 krb5kdc[24339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:26 dist-test-slave-wmg0 krb5kdc[24339](info): set up 2 sockets
Sep 27 10:47:26 dist-test-slave-wmg0 krb5kdc[24339](info): commencing operation
krb5kdc: starting...
W20260927 10:47:27.521706 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:47:27 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:47:27.648919 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33485
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:27.761008 24352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:27.761281 24352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:27.761322 24352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:27.765638 24352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:27.765712 24352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:27.765731 24352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:27.765749 24352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:27.765766 24352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:27.771945 24352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:27.773121 24352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:27.774255 24352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:27.783301 24358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:27.783303 24357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:27.783301 24360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:27.783674 24352 server_base.cc:1034] running on GCE node
I20260927 10:47:27.784267 24352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:27.785529 24352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:27.786774 24352 hybrid_clock.cc:648] HybridClock initialized: now 1790506047786757 us; error 29 us; skew 500 ppm
Sep 27 10:47:27 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506047, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:27.795890 24352 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:27.796945 24352 webserver.cc:460] Webserver started at http://127.10.96.254:45051/ using document root <none> and password file <none>
I20260927 10:47:27.797631 24352 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:27.797730 24352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:27.797953 24352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:27.800122 24352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "2e7e801fed454d07a7932227a6abf5bf"
format_stamp: "Formatted at 2026-09-27 10:47:27 on dist-test-slave-wmg0"
server_key: "bc8a9ec185889635069e926109279942"
server_key_iv: "f2bfdcbf5368ce2b6a116c3856d7fea2"
server_key_version: "encryptionkey@0"
I20260927 10:47:27.800755 24352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "2e7e801fed454d07a7932227a6abf5bf"
format_stamp: "Formatted at 2026-09-27 10:47:27 on dist-test-slave-wmg0"
server_key: "bc8a9ec185889635069e926109279942"
server_key_iv: "f2bfdcbf5368ce2b6a116c3856d7fea2"
server_key_version: "encryptionkey@0"
I20260927 10:47:27.804672 24352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:27.807320 24367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:27.808713 24352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:47:27.808878 24352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "2e7e801fed454d07a7932227a6abf5bf"
format_stamp: "Formatted at 2026-09-27 10:47:27 on dist-test-slave-wmg0"
server_key: "bc8a9ec185889635069e926109279942"
server_key_iv: "f2bfdcbf5368ce2b6a116c3856d7fea2"
server_key_version: "encryptionkey@0"
I20260927 10:47:27.809002 24352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:27.829538 24352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:27.833069 24352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:27.833300 24352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:27.840739 24352 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:33485
I20260927 10:47:27.841073 24419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:33485 every 8 connection(s)
I20260927 10:47:27.841853 24352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:27.844784 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:27.846061 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24352
I20260927 10:47:27.846213 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:47:27.846588 10627 external_mini_cluster.cc:1428] Setting key 96a0b4ebafa2bc1f2cb4b84b230db368
I20260927 10:47:27.851555 24420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap starting.
I20260927 10:47:27.854095 24420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:27.854952 24420 log.cc:824] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:47:27 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:27.857718 24420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: No bootstrap required, opened a new log
I20260927 10:47:27.861429 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:27.848162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.10.96.254:33485)
0927 10:47:27.848612 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:27.848626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:27.848661 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:47:27.849290 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:27.849306 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:27.849708 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:27.850033 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:27.850822 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:27.851847 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:27.852714 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:27.853188 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:27.853371 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:27.856315 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:27.856321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:27.856338 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:27.856368 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:27.858402 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:27.858836 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:27.858842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:27.858848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:27.858973 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:27.859223 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:27.859226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:27.859228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:27.859762 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:27.859994 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:27.860188 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 10:47:27.860489 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260927 10:47:27.862228 24420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:27.862484 24420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:27.862534 24420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7e801fed454d07a7932227a6abf5bf, State: Initialized, Role: FOLLOWER
I20260927 10:47:27.862991 24420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:27.863157 24420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:27.863225 24420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:27.863322 24420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:27.864463 24420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:27.864785 24420 leader_election.cc:302] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e7e801fed454d07a7932227a6abf5bf; no voters:
I20260927 10:47:27.865052 24420 leader_election.cc:288] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:27.865185 24425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:27.865427 24425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 LEADER]: Becoming Leader. State: Replica: 2e7e801fed454d07a7932227a6abf5bf, State: Running, Role: LEADER
I20260927 10:47:27.865828 24425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:27.866326 24420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:27.867466 24427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e7e801fed454d07a7932227a6abf5bf. Latest consensus state: current_term: 1 leader_uuid: "2e7e801fed454d07a7932227a6abf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } } }
I20260927 10:47:27.867599 24427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:27.867857 24426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e7e801fed454d07a7932227a6abf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } } }
I20260927 10:47:27.867959 24434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:27.868052 24426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:27.871196 24434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:27.876852 24434 catalog_manager.cc:1347] Generated new cluster ID: 92a7f7cce9824ca0bac7354ff0c6ca3e
I20260927 10:47:27.876922 24434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:27.892421 24434 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:27.894012 24434 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:27.900242 24434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Generated new TSK 0
I20260927 10:47:27.901059 24434 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:47:27.930351 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33485
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:28.046649 24446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:28.046932 24446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:28.046976 24446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:28.051128 24446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:28.051208 24446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:28.051270 24446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:47:28.057391 24446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:28.058624 24446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:28.059708 24446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:28.069677 24446 server_base.cc:1034] running on GCE node
W20260927 10:47:28.069628 24452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.069629 24454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.069638 24451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:28.070340 24446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:28.070928 24446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:28.072124 24446 hybrid_clock.cc:648] HybridClock initialized: now 1790506048072107 us; error 30 us; skew 500 ppm
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:28.075023 24446 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:47:28.075831 24446 webserver.cc:460] Webserver started at http://127.10.96.193:40051/ using document root <none> and password file <none>
I20260927 10:47:28.076366 24446 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:28.076424 24446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:28.076654 24446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:28.078729 24446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "0a5f83790299cf319c13d74f6b14b526"
server_key_iv: "1fef37ff708abb40e02e81d35e172443"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.079288 24446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "0a5f83790299cf319c13d74f6b14b526"
server_key_iv: "1fef37ff708abb40e02e81d35e172443"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.083024 24446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:28.085712 24461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.086982 24446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:28.087111 24446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "0a5f83790299cf319c13d74f6b14b526"
server_key_iv: "1fef37ff708abb40e02e81d35e172443"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.087215 24446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:28.099128 24446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:28.102555 24446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:28.102764 24446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:28.103423 24446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:28.104169 24446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:28.104218 24446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.104256 24446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:28.104280 24446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.112895 24446 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:35843
I20260927 10:47:28.112967 24574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:35843 every 8 connection(s)
I20260927 10:47:28.113976 24446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:28.117185 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24446
I20260927 10:47:28.117354 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:28.117700 10627 external_mini_cluster.cc:1428] Setting key 2075a95328b3e51bb639fd65413e9f0c
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:47:28.130105 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.115837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:42393 (local address 127.10.96.254:33485)
0927 10:47:28.115986 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.115990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.117016 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 10:47:28.118895 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.118897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.118940 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.119032 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:28.120332 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.121091 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.122935 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.123155 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.123236 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.126227 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:28.126231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:28.126234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:28.126257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:28.127876 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.128384 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.128387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.128389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.128470 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.128847 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.128850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.128852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.129031 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:28.129135 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.129750 (+ 615us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.129902 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:47:28.131121 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.116209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:42393)
0927 10:47:28.116744 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.116792 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.118658 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.119229 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.119240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.119654 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:28.120177 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.120190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.121233 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.121240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.122813 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.122828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.123334 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.123339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.123597 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.124248 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:28.124275 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:28.126047 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:28.127992 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:28.127997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:28.128013 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:28.128268 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:28.128571 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:28.128573 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:28.128575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:28.128741 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:28.129145 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:28.129153 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:28.129449 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.129696 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.130030 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260927 10:47:28.132416 24575 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33485
I20260927 10:47:28.132692 24575 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:28.133277 24575 heartbeater.cc:511] Master 127.10.96.254:33485 requested a full tablet report, sending...
I20260927 10:47:28.134902 24384 ts_manager.cc:194] Registered new tserver with Master: 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843)
I20260927 10:47:28.136018 24384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:42393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:47:28.146111 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33485
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:28.260545 24580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:28.260888 24580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:28.260969 24580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:28.265287 24580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:28.265391 24580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:28.265540 24580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:47:28.271189 24580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:28.272280 24580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:28.273344 24580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:28.283459 24580 server_base.cc:1034] running on GCE node
W20260927 10:47:28.283452 24588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.283450 24585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.283444 24586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:28.284076 24580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:28.284672 24580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:28.285859 24580 hybrid_clock.cc:648] HybridClock initialized: now 1790506048285824 us; error 50 us; skew 500 ppm
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:28.288851 24580 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:47:28.289829 24580 webserver.cc:460] Webserver started at http://127.10.96.194:42829/ using document root <none> and password file <none>
I20260927 10:47:28.290445 24580 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:28.290508 24580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:28.290719 24580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:28.292809 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "a0996ba87c484dba89b3c6aac854347f"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "8125e3e3115874280bb09bb143c2241e"
server_key_iv: "133716ce437b3f9e2339764ad243f2c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.293380 24580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "a0996ba87c484dba89b3c6aac854347f"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "8125e3e3115874280bb09bb143c2241e"
server_key_iv: "133716ce437b3f9e2339764ad243f2c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.297173 24580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:28.299886 24595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.301163 24580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 10:47:28.301311 24580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a0996ba87c484dba89b3c6aac854347f"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "8125e3e3115874280bb09bb143c2241e"
server_key_iv: "133716ce437b3f9e2339764ad243f2c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.301433 24580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:28.320045 24580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:28.323551 24580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:28.323765 24580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:28.324393 24580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:28.325248 24580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:28.325294 24580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.325361 24580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:28.325392 24580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.334499 24580 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:46087
I20260927 10:47:28.334523 24708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:46087 every 8 connection(s)
I20260927 10:47:28.335389 24580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:28.342944 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24580
I20260927 10:47:28.343077 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:28.343443 10627 external_mini_cluster.cc:1428] Setting key ab0fc9c93b725e02219ab19b69e80e34
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:28.349761 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.337242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36263 (local address 127.10.96.254:33485)
0927 10:47:28.337392 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.337397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.338358 (+ 961us) server_negotiation.cc:407] Connection header received
0927 10:47:28.339664 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.339667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.339714 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.339796 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:28.340938 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.341526 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.342705 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.343051 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.343169 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.345974 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:28.345981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:28.345987 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:28.346019 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:28.347555 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.348111 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.348117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.348122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.348224 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.348546 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.348553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.348557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.348814 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:28.348923 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.349388 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.349545 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:47:28.350569 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.337732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:36263)
0927 10:47:28.338180 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.338243 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.339425 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.339955 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.339961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.340329 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:28.340793 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.340806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.341672 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.341674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.342558 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.342566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.343227 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.343231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.343379 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.344109 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:28.344130 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:28.345805 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:28.347668 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:28.347675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:28.347690 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:28.348000 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:28.348320 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:28.348322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:28.348324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:28.348452 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:28.348916 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:28.348920 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:28.349143 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.349305 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.349637 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260927 10:47:28.351737 24709 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33485
I20260927 10:47:28.352005 24709 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:28.352609 24709 heartbeater.cc:511] Master 127.10.96.254:33485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:47:28.353886 24384 ts_manager.cc:194] Registered new tserver with Master: a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
I20260927 10:47:28.354468 24384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:36263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:47:28.372051 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33485
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:28.483990 24714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:28.484241 24714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:28.484283 24714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:28.488355 24714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:28.488420 24714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:28.488478 24714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:47:28.494005 24714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:28.495076 24714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:28.496070 24714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:28.506177 24720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:28.506242 24714 server_base.cc:1034] running on GCE node
W20260927 10:47:28.506181 24722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.506191 24719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:28.506877 24714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:28.507572 24714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:28.508756 24714 hybrid_clock.cc:648] HybridClock initialized: now 1790506048508739 us; error 30 us; skew 500 ppm
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:28.511538 24714 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:47:28.512547 24714 webserver.cc:460] Webserver started at http://127.10.96.195:43917/ using document root <none> and password file <none>
I20260927 10:47:28.513123 24714 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:28.513185 24714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:28.513422 24714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:28.515564 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "10b8589209b7404fa369524b048fc027"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "3a8d4dc7873aff5da856508cd21c8c6a"
server_key_iv: "e521de4d0fc598b83c9000216145bd2a"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.516073 24714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "10b8589209b7404fa369524b048fc027"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "3a8d4dc7873aff5da856508cd21c8c6a"
server_key_iv: "e521de4d0fc598b83c9000216145bd2a"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.519776 24714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 10:47:28.522183 24729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.523519 24714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:47:28.523658 24714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "10b8589209b7404fa369524b048fc027"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "3a8d4dc7873aff5da856508cd21c8c6a"
server_key_iv: "e521de4d0fc598b83c9000216145bd2a"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.523773 24714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:28.543201 24714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:28.546381 24714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:28.546590 24714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:28.547230 24714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:28.548095 24714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:28.548171 24714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.548243 24714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:28.548292 24714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.557493 24714 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40473
I20260927 10:47:28.557560 24842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40473 every 8 connection(s)
I20260927 10:47:28.558461 24714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:28.568645 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24714
I20260927 10:47:28.568828 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:28.569164 10627 external_mini_cluster.cc:1428] Setting key 10a767edad10d577827c7aa6f836a640
I20260927 10:47:28.572346 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.560348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:50475 (local address 127.10.96.254:33485)
0927 10:47:28.560509 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.560513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.561431 (+ 918us) server_negotiation.cc:407] Connection header received
0927 10:47:28.562856 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.562858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.562903 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.562980 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:28.564083 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.564609 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.565731 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.566036 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.566139 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.568757 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:28.568760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:28.568763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:28.568788 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:28.570422 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.570926 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.570930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.570932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.571023 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.571411 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.571414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.571415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.571595 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:28.571678 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.572141 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.572234 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:47:28.573182 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.560818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:50475)
0927 10:47:28.561274 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.561315 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.562645 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.563133 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.563139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.563500 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:28.563966 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.563979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.564713 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.564716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.565595 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.565603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.566168 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.566170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.566324 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.566943 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:28.566966 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:28.568551 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:28.570527 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:28.570534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:28.570548 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:28.570809 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:28.571141 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:28.571146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:28.571148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:28.571328 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:28.571689 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:28.571726 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:28.571946 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.572098 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.572337 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260927 10:47:28.574358 24843 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33485
I20260927 10:47:28.574605 24843 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:28.575078 24843 heartbeater.cc:511] Master 127.10.96.254:33485 requested a full tablet report, sending...
I20260927 10:47:28.576035 24384 ts_manager.cc:194] Registered new tserver with Master: 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473)
I20260927 10:47:28.576629 24384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:50475
I20260927 10:47:28.582870 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:28.592338 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.584735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.10.96.254:33485)
0927 10:47:28.584879 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.584883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.585090 (+ 207us) server_negotiation.cc:407] Connection header received
0927 10:47:28.585242 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.585245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.585289 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.585379 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:28.586227 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.586738 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.587583 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.587825 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.587929 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.589253 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:28.589256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:28.589259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:28.589288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:28.590579 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.591052 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.591057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.591059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.591137 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:28.591419 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:28.591422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:28.591424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:28.591635 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:28.591799 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.592051 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.592170 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:47:28.594995 24384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:28.598114 24384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:28.612664 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.607271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35843 (local address 127.0.0.1:48714)
0927 10:47:28.607745 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.607760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.607833 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.608730 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.608733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.608749 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:28.608837 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.608842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.610205 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.610209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.611013 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.611022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.612129 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.612132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.612243 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.612290 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.612402 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.612539 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260927 10:47:28.613240 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.606641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:46087 (local address 127.0.0.1:42562)
0927 10:47:28.607476 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.607491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.607575 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.608321 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.608324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.608341 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:28.608433 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.608438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.609877 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.609880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.611255 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.611265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.612857 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.612861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.612943 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.612964 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.613040 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.613107 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":59,"thread_start_us":134,"threads_started":1}
I20260927 10:47:28.613336 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.607221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.10.96.194:46087)
0927 10:47:28.607568 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.607579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.607599 (+ 20us) server_negotiation.cc:407] Connection header received
0927 10:47:28.607763 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.607770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.607994 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.608155 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:28.608568 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.609743 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.611398 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.612186 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.612278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.612414 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.613074 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.613171 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":87,"threads_started":1}
I20260927 10:47:28.613736 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.607673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.10.96.193:35843)
0927 10:47:28.608151 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.608165 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.608186 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:47:28.608257 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.608262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.608439 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.608574 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:28.608967 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.610068 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.611173 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.612008 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.612134 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.612288 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.613419 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.613559 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":64,"threads_started":1}
I20260927 10:47:28.614568 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.606553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40473 (local address 127.0.0.1:37176)
0927 10:47:28.607059 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.607107 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.607235 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.607829 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.607836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.607857 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:28.608037 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.608045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.609711 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.609717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.611254 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.611265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.613475 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.613495 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.613584 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.613609 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.613662 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.613749 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":117,"thread_start_us":74,"threads_started":1}
I20260927 10:47:28.615152 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.606848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.10.96.195:40473)
0927 10:47:28.607287 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.607301 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.607329 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:47:28.607406 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.607411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.607646 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.608206 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:28.608388 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.609589 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.612365 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.613325 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.614744 (+ 1419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.614873 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.614922 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.615005 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":151,"threads_started":1}
I20260927 10:47:28.616523 24643 tablet_service.cc:1467] Processing CreateTablet for tablet 85d7210f77814e34a1ab333775a51336 (DEFAULT_TABLE table=test-table [id=3221afe70305421aa822a3b9ed8c72da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:28.616521 24509 tablet_service.cc:1467] Processing CreateTablet for tablet 85d7210f77814e34a1ab333775a51336 (DEFAULT_TABLE table=test-table [id=3221afe70305421aa822a3b9ed8c72da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:28.617427 24643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d7210f77814e34a1ab333775a51336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:28.617540 24509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d7210f77814e34a1ab333775a51336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:28.617447 24777 tablet_service.cc:1467] Processing CreateTablet for tablet 85d7210f77814e34a1ab333775a51336 (DEFAULT_TABLE table=test-table [id=3221afe70305421aa822a3b9ed8c72da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:28.618345 24777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d7210f77814e34a1ab333775a51336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:28.623450 24860 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Bootstrap starting.
I20260927 10:47:28.625296 24861 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap starting.
I20260927 10:47:28.625775 24862 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Bootstrap starting.
I20260927 10:47:28.626296 24860 tablet_bootstrap.cc:654] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:28.626992 24860 log.cc:824] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:28.627386 24861 tablet_bootstrap.cc:654] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:28.627712 24862 tablet_bootstrap.cc:654] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:28.628743 24861 log.cc:824] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:28.628859 24862 log.cc:824] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:28.629071 24860 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: No bootstrap required, opened a new log
I20260927 10:47:28.629290 24860 ts_tablet_manager.cc:1397] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:28.630704 24861 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: No bootstrap required, opened a new log
I20260927 10:47:28.630903 24861 ts_tablet_manager.cc:1397] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:28.630976 24862 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: No bootstrap required, opened a new log
I20260927 10:47:28.631155 24862 ts_tablet_manager.cc:1397] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:28.633078 24860 raft_consensus.cc:348] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.633352 24860 raft_consensus.cc:374] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:28.633416 24860 raft_consensus.cc:729] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0996ba87c484dba89b3c6aac854347f, State: Initialized, Role: FOLLOWER
I20260927 10:47:28.633890 24860 consensus_queue.cc:260] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.633862 24861 raft_consensus.cc:348] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.634088 24861 raft_consensus.cc:374] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:28.634141 24861 raft_consensus.cc:729] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b2f3fd3ed455da89e9f5045d0692e, State: Initialized, Role: FOLLOWER
I20260927 10:47:28.634586 24709 heartbeater.cc:503] Master 127.10.96.254:33485 was elected leader, sending a full tablet report...
I20260927 10:47:28.634563 24861 consensus_queue.cc:260] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.634887 24860 ts_tablet_manager.cc:1428] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:28.634845 24862 raft_consensus.cc:348] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.635062 24862 raft_consensus.cc:374] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:28.635116 24862 raft_consensus.cc:729] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b8589209b7404fa369524b048fc027, State: Initialized, Role: FOLLOWER
I20260927 10:47:28.635442 24861 ts_tablet_manager.cc:1428] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 10:47:28.635567 24575 heartbeater.cc:503] Master 127.10.96.254:33485 was elected leader, sending a full tablet report...
I20260927 10:47:28.635519 24862 consensus_queue.cc:260] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.636430 24862 ts_tablet_manager.cc:1428] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:28.636765 24843 heartbeater.cc:503] Master 127.10.96.254:33485 was elected leader, sending a full tablet report...
I20260927 10:47:28.736043 24866 raft_consensus.cc:482] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:28.736246 24866 raft_consensus.cc:504] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.737389 24866 leader_election.cc:288] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843)
I20260927 10:47:28.741278 24856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.737724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:46247 (local address 127.10.96.195:40473)
0927 10:47:28.737865 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.737869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.738224 (+ 355us) server_negotiation.cc:407] Connection header received
0927 10:47:28.738345 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.738348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.738394 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.738475 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:28.738768 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.739257 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.740299 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.740704 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.740793 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.740867 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.741071 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.741147 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:47:28.741288 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.737656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35843 (local address 127.10.96.194:57987)
0927 10:47:28.737910 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.737925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.738032 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.738330 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.738333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.738350 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:28.738488 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.738495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.739301 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.739307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.740167 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.740175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.740889 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.740894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.740990 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.741006 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.741103 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.741165 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":51}
I20260927 10:47:28.741297 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.737659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40473 (local address 127.10.96.194:46247)
0927 10:47:28.738115 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.738128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:28.738236 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:28.738494 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:28.738496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:28.738510 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:28.738657 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.738665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.739374 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.739376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.740164 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:28.740173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.740828 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.740835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:28.740926 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.740959 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 10:47:28.741030 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 10:47:28.741116 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":38,"thread_start_us":125,"threads_started":1}
I20260927 10:47:28.741321 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.737762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:57987 (local address 127.10.96.193:35843)
0927 10:47:28.737894 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.737899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.738007 (+ 108us) server_negotiation.cc:407] Connection header received
0927 10:47:28.738142 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.738148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.738214 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.738296 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:28.738602 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.739190 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.740337 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.740738 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.740876 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.740928 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.741140 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.741215 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:47:28.741993 24529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d7210f77814e34a1ab333775a51336" candidate_uuid: "a0996ba87c484dba89b3c6aac854347f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" is_pre_election: true
I20260927 10:47:28.742007 24797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d7210f77814e34a1ab333775a51336" candidate_uuid: "a0996ba87c484dba89b3c6aac854347f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b8589209b7404fa369524b048fc027" is_pre_election: true
I20260927 10:47:28.742288 24529 raft_consensus.cc:2454] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0996ba87c484dba89b3c6aac854347f in term 0.
I20260927 10:47:28.742300 24797 raft_consensus.cc:2454] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0996ba87c484dba89b3c6aac854347f in term 0.
I20260927 10:47:28.742791 24599 leader_election.cc:302] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f; no voters:
I20260927 10:47:28.743063 24866 raft_consensus.cc:2781] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:28.743146 24866 raft_consensus.cc:482] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:28.743196 24866 raft_consensus.cc:3037] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:28.744179 24866 raft_consensus.cc:504] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.744525 24866 leader_election.cc:288] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [CANDIDATE]: Term 1 election: Requested vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843)
I20260927 10:47:28.744843 24797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d7210f77814e34a1ab333775a51336" candidate_uuid: "a0996ba87c484dba89b3c6aac854347f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b8589209b7404fa369524b048fc027"
I20260927 10:47:28.744913 24529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d7210f77814e34a1ab333775a51336" candidate_uuid: "a0996ba87c484dba89b3c6aac854347f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
I20260927 10:47:28.744994 24797 raft_consensus.cc:3037] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:28.745055 24529 raft_consensus.cc:3037] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:28.746119 24797 raft_consensus.cc:2454] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0996ba87c484dba89b3c6aac854347f in term 1.
I20260927 10:47:28.746235 24529 raft_consensus.cc:2454] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0996ba87c484dba89b3c6aac854347f in term 1.
I20260927 10:47:28.746380 24599 leader_election.cc:302] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f; no voters:
I20260927 10:47:28.746557 24866 raft_consensus.cc:2781] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:28.746794 24866 raft_consensus.cc:686] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 1 LEADER]: Becoming Leader. State: Replica: a0996ba87c484dba89b3c6aac854347f, State: Running, Role: LEADER
I20260927 10:47:28.747133 24866 consensus_queue.cc:237] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:28.750221 24382 catalog_manager.cc:5495] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f reported cstate change: term changed from 0 to 1, leader changed from <none> to a0996ba87c484dba89b3c6aac854347f (127.10.96.194). New cstate: current_term: 1 leader_uuid: "a0996ba87c484dba89b3c6aac854347f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } health_report { overall_health: UNKNOWN } } }
I20260927 10:47:28.766222 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.762666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.10.96.194:46087)
0927 10:47:28.762794 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:28.762797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:28.762813 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:47:28.762971 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:28.762974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:28.763029 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:28.763131 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:28.763313 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.763824 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.764738 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:28.764951 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:28.765039 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:28.765306 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:28.765813 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:28.765909 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:28.766020 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 10:47:28.766071 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:47:28.772259 24797 raft_consensus.cc:1261] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Refusing update from remote peer a0996ba87c484dba89b3c6aac854347f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:28.772259 24529 raft_consensus.cc:1261] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Refusing update from remote peer a0996ba87c484dba89b3c6aac854347f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:28.773049 24866 consensus_queue.cc:1035] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:28.773232 24870 consensus_queue.cc:1035] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:28.781677 24875 mvcc.cc:205] Tried to move back new op lower bound from 7333912775763308544 to 7333912775670960128. Current Snapshot: MvccSnapshot[applied={T|T < 7333912775763308544}]
I20260927 10:47:28.782349 24874 mvcc.cc:205] Tried to move back new op lower bound from 7333912775763308544 to 7333912775670960128. Current Snapshot: MvccSnapshot[applied={T|T < 7333912775763308544}]
I20260927 10:47:28.790241 24382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:28.790423 24382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:28.793052 24382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Sending DeleteTablet for 3 replicas of tablet 85d7210f77814e34a1ab333775a51336
I20260927 10:47:28.793869 24777 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d7210f77814e34a1ab333775a51336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:28 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:37176
I20260927 10:47:28.793951 24643 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d7210f77814e34a1ab333775a51336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:28 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:42562
I20260927 10:47:28.793951 24509 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d7210f77814e34a1ab333775a51336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:28 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:48714
I20260927 10:47:28.794435 24883 tablet_replica.cc:330] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: stopping tablet replica
I20260927 10:47:28.794561 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24446
I20260927 10:47:28.795395 24884 tablet_replica.cc:330] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: stopping tablet replica
I20260927 10:47:28.795722 24884 raft_consensus.cc:2229] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:28.796101 24882 tablet_replica.cc:330] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: stopping tablet replica
I20260927 10:47:28.796140 24884 raft_consensus.cc:2258] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:28.796397 24882 raft_consensus.cc:2229] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:28.796718 24882 raft_consensus.cc:2258] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:28.798337 24882 ts_tablet_manager.cc:1905] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:28.801061 24882 ts_tablet_manager.cc:1918] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:28.801189 24882 log.cc:1196] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/85d7210f77814e34a1ab333775a51336
I20260927 10:47:28.801537 24882 ts_tablet_manager.cc:1939] T 85d7210f77814e34a1ab333775a51336 P 10b8589209b7404fa369524b048fc027: Deleting consensus metadata
I20260927 10:47:28.802126 24884 ts_tablet_manager.cc:1905] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:28.802753 24371 catalog_manager.cc:4841] TS 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): tablet 85d7210f77814e34a1ab333775a51336 (table test-table [id=3221afe70305421aa822a3b9ed8c72da]) successfully deleted
I20260927 10:47:28.804351 24884 ts_tablet_manager.cc:1918] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:28.804482 24884 log.cc:1196] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/85d7210f77814e34a1ab333775a51336
I20260927 10:47:28.804823 24884 ts_tablet_manager.cc:1939] T 85d7210f77814e34a1ab333775a51336 P a0996ba87c484dba89b3c6aac854347f: Deleting consensus metadata
W20260927 10:47:28.806025 24368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:35843 (error 108)
I20260927 10:47:28.806289 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24580
I20260927 10:47:28.806548 24368 catalog_manager.cc:4841] TS a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087): tablet 85d7210f77814e34a1ab333775a51336 (table test-table [id=3221afe70305421aa822a3b9ed8c72da]) successfully deleted
I20260927 10:47:28.807981 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:28.806831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35843 (local address 127.0.0.1:48718)
0927 10:47:28.807791 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:28.807875 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:35843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":821}
W20260927 10:47:28.808161 24368 catalog_manager.cc:4584] TS 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85d7210f77814e34a1ab333775a51336: Network error: Client connection negotiation failed: client connection to 127.10.96.193:35843: connect: Connection refused (error 111)
I20260927 10:47:28.812803 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24714
I20260927 10:47:28.818094 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24352
I20260927 10:47:28.824270 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33485
--webserver_interface=127.10.96.254
--webserver_port=45051
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:28.933853 24886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:28.934109 24886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:28.934149 24886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:28.938151 24886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:28.938212 24886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:28.938231 24886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:28.938247 24886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:28.938263 24886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:28.943557 24886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=45051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:28.944645 24886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:28.945719 24886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:28.958709 24895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.958699 24892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:28.958699 24893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:28.958745 24886 server_base.cc:1034] running on GCE node
I20260927 10:47:28.959542 24886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:28.960726 24886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:28.961930 24886 hybrid_clock.cc:648] HybridClock initialized: now 1790506048961893 us; error 50 us; skew 500 ppm
Sep 27 10:47:28 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506048, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:28.964713 24886 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:28.965775 24886 webserver.cc:460] Webserver started at http://127.10.96.254:45051/ using document root <none> and password file <none>
I20260927 10:47:28.966393 24886 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:28.966509 24886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:28.970570 24886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:28.972875 24902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:28.974255 24886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:47:28.974411 24886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "2e7e801fed454d07a7932227a6abf5bf"
format_stamp: "Formatted at 2026-09-27 10:47:27 on dist-test-slave-wmg0"
server_key: "bc8a9ec185889635069e926109279942"
server_key_iv: "f2bfdcbf5368ce2b6a116c3856d7fea2"
server_key_version: "encryptionkey@0"
I20260927 10:47:28.975030 24886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:28.990142 24886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:28.993352 24886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:28.993597 24886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:29.000969 24886 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:33485
I20260927 10:47:29.001379 24954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:33485 every 8 connection(s)
I20260927 10:47:29.002254 24886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:29.006791 24955 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:47:29.008440 24955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap starting.
I20260927 10:47:29.010720 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24886
I20260927 10:47:29.011132 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:35843
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:33485
--webserver_port=40051
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:29.025377 24955 log.cc:824] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:29.034214 24955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:29.034614 24955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap complete.
I20260927 10:47:29.038827 24955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:29.039268 24955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7e801fed454d07a7932227a6abf5bf, State: Initialized, Role: FOLLOWER
I20260927 10:47:29.040134 24955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:29.040333 24955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:29.040434 24955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:29.040555 24955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:47:29.042797 24955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:29.043243 24955 leader_election.cc:302] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e7e801fed454d07a7932227a6abf5bf; no voters:
I20260927 10:47:29.043646 24955 leader_election.cc:288] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:47:29.043733 24959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:47:29.043931 24959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 LEADER]: Becoming Leader. State: Replica: 2e7e801fed454d07a7932227a6abf5bf, State: Running, Role: LEADER
I20260927 10:47:29.044247 24959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:29.044957 24955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:29.045627 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e7e801fed454d07a7932227a6abf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } } }
I20260927 10:47:29.045759 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:29.046454 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e7e801fed454d07a7932227a6abf5bf. Latest consensus state: current_term: 2 leader_uuid: "2e7e801fed454d07a7932227a6abf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } } }
I20260927 10:47:29.046617 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:29.046859 24966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:29.052069 24966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:29.052780 24966 catalog_manager.cc:1259] Loaded cluster ID: 92a7f7cce9824ca0bac7354ff0c6ca3e
I20260927 10:47:29.052860 24966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:29.054327 24966 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 10:47:29.058918 24966 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 10:47:29.064860 24966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:29.065759 24966 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Loaded TSK: 0
I20260927 10:47:29.067006 24966 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:47:29.135952 24957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:29.136221 24957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:29.136268 24957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:29.140460 24957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:29.140527 24957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:29.140588 24957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:47:29.146106 24957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:35843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=40051
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:29.147233 24957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:29.148274 24957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:29.158871 24983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.158805 24982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:29.158981 24957 server_base.cc:1034] running on GCE node
W20260927 10:47:29.158807 24985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:29.159690 24957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:29.160494 24957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:29.161688 24957 hybrid_clock.cc:648] HybridClock initialized: now 1790506049161661 us; error 36 us; skew 500 ppm
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:29.165181 24957 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:47:29.166388 24957 webserver.cc:460] Webserver started at http://127.10.96.193:40051/ using document root <none> and password file <none>
I20260927 10:47:29.167094 24957 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:29.167218 24957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:29.172250 24957 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:47:29.174896 24992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.176326 24957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:47:29.176435 24957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "0a5f83790299cf319c13d74f6b14b526"
server_key_iv: "1fef37ff708abb40e02e81d35e172443"
server_key_version: "encryptionkey@0"
I20260927 10:47:29.177109 24957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:29.210498 24957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:29.213521 24957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:29.213676 24957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:29.214241 24957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:29.215821 24999 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:47:29.219990 24957 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:47:29.220057 24957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 10:47:29.220110 24957 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:47:29.221836 24957 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:47:29.221884 24957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:29.222273 24999 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap starting.
I20260927 10:47:29.232335 24957 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:35843
I20260927 10:47:29.232407 25106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:35843 every 8 connection(s)
I20260927 10:47:29.233348 24957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:29.238229 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 24957
I20260927 10:47:29.238739 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:46087
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:33485
--webserver_port=42829
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:29.241343 24999 log.cc:824] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:29.249534 24999 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:29.249922 24999 tablet_bootstrap.cc:492] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap complete.
I20260927 10:47:29.251272 24999 ts_tablet_manager.cc:1397] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20260927 10:47:29.253906 25110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.235607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:49751)
0927 10:47:29.236318 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.236383 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.237554 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.238817 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.238825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.239399 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:29.239864 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.239878 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.241822 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.241824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.242967 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.242976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.244015 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.244023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.244259 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.245614 (+ 1355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:29.245634 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:29.247355 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:29.250365 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:29.250374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:29.250390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:29.250779 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:29.251231 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:29.251234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:29.251236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:29.251401 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:29.252105 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:29.252112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:29.252387 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.252596 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.252932 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":170,"threads_started":1}
I20260927 10:47:29.254271 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.235636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:49751 (local address 127.10.96.254:33485)
0927 10:47:29.236041 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.236060 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.236518 (+ 458us) server_negotiation.cc:407] Connection header received
0927 10:47:29.238012 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.238024 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.238366 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.238723 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:29.240041 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.241660 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.243148 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.243851 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.244023 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.247606 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:29.247616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:29.247636 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:29.247685 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:29.250215 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.250908 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.250918 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.250927 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.251114 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.251507 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.251510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.251512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.251976 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:29.252107 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.252733 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.253029 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260927 10:47:29.254082 24999 raft_consensus.cc:348] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.254614 24999 raft_consensus.cc:729] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b2f3fd3ed455da89e9f5045d0692e, State: Initialized, Role: FOLLOWER
I20260927 10:47:29.255079 24999 consensus_queue.cc:260] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.255992 24999 ts_tablet_manager.cc:1428] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:29.256698 25107 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33485
I20260927 10:47:29.256950 25107 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:29.257509 25107 heartbeater.cc:511] Master 127.10.96.254:33485 requested a full tablet report, sending...
I20260927 10:47:29.260277 24919 ts_manager.cc:194] Registered new tserver with Master: 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843)
I20260927 10:47:29.264353 24919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:49751
I20260927 10:47:29.267879 25107 heartbeater.cc:503] Master 127.10.96.254:33485 was elected leader, sending a full tablet report...
I20260927 10:47:29.273147 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.264234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.10.96.193:35843)
0927 10:47:29.264684 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.264697 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.268772 (+ 4075us) server_negotiation.cc:407] Connection header received
0927 10:47:29.268844 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.268849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.269042 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.269214 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:29.269676 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.270724 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.271925 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.272444 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.272568 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.272707 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.272910 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.273002 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260927 10:47:29.273147 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.263604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35843 (local address 127.0.0.1:48720)
0927 10:47:29.267273 (+ 3669us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.268471 (+ 1198us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.268626 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.269360 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.269366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.269389 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:29.269543 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.269550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.270891 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.270895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.271786 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.271796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.272574 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.272577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.272702 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.272749 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.272869 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.273002 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3486,"thread_start_us":102,"threads_started":1}
I20260927 10:47:29.274233 25041 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d7210f77814e34a1ab333775a51336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:28 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:48720
I20260927 10:47:29.275128 25116 tablet_replica.cc:330] stopping tablet replica
I20260927 10:47:29.275460 25116 raft_consensus.cc:2229] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:29.275722 25116 raft_consensus.cc:2258] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:29.276818 25116 ts_tablet_manager.cc:1905] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:29.281546 25116 ts_tablet_manager.cc:1918] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:29.281658 25116 log.cc:1196] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/85d7210f77814e34a1ab333775a51336
I20260927 10:47:29.281936 25116 ts_tablet_manager.cc:1939] T 85d7210f77814e34a1ab333775a51336 P 4e9b2f3fd3ed455da89e9f5045d0692e: Deleting consensus metadata
I20260927 10:47:29.284026 24903 catalog_manager.cc:4841] TS 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843): tablet 85d7210f77814e34a1ab333775a51336 (table test-table [id=3221afe70305421aa822a3b9ed8c72da]) successfully deleted
W20260927 10:47:29.365085 25111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:29.365339 25111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:29.365381 25111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:29.369524 25111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:29.369585 25111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:29.369648 25111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:47:29.375206 25111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=42829
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:29.376420 25111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:29.377537 25111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:29.387758 25124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.387784 25122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.387789 25121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:29.387988 25111 server_base.cc:1034] running on GCE node
I20260927 10:47:29.388382 25111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:29.388993 25111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:29.390147 25111 hybrid_clock.cc:648] HybridClock initialized: now 1790506049390116 us; error 43 us; skew 500 ppm
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:29.393038 25111 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:47:29.393970 25111 webserver.cc:460] Webserver started at http://127.10.96.194:42829/ using document root <none> and password file <none>
I20260927 10:47:29.394497 25111 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:29.394570 25111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:29.398452 25111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:29.400456 25131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.401759 25111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:47:29.401858 25111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a0996ba87c484dba89b3c6aac854347f"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "8125e3e3115874280bb09bb143c2241e"
server_key_iv: "133716ce437b3f9e2339764ad243f2c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:29.402503 25111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:29.418108 25111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:29.421258 25111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:29.421552 25111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:29.422184 25111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:29.423041 25111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:29.423112 25111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.423187 25111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:29.423245 25111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.431859 25111 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:46087
I20260927 10:47:29.431902 25244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:46087 every 8 connection(s)
I20260927 10:47:29.432904 25111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:29.436300 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25111
I20260927 10:47:29.436856 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:40473
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:33485
--webserver_port=43917
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:29.448706 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.434766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:37309 (local address 127.10.96.254:33485)
0927 10:47:29.434938 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.434943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.435822 (+ 879us) server_negotiation.cc:407] Connection header received
0927 10:47:29.437827 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.437830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.437880 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.437963 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:29.439328 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.440220 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.441487 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.441955 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.442091 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.444503 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:29.444509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:29.444514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:29.444551 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:29.446496 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.447078 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.447084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.447088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.447193 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.447537 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.447542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.447545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.447823 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:29.447943 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.448389 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.448533 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 10:47:29.449539 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.435222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:37309)
0927 10:47:29.435672 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.435715 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.437572 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.438180 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.438192 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.438681 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:29.439161 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.439175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.440357 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.440359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.441312 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.441321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.442110 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.442112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.442251 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.442888 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:29.442903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:29.444316 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:29.446616 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:29.446620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:29.446632 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:29.446975 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:29.447298 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:29.447300 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:29.447302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:29.447442 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:29.447929 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:29.447934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:29.448161 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.448317 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.448578 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260927 10:47:29.450802 25245 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33485
I20260927 10:47:29.451073 25245 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:29.451567 25245 heartbeater.cc:511] Master 127.10.96.254:33485 requested a full tablet report, sending...
I20260927 10:47:29.452613 24919 ts_manager.cc:194] Registered new tserver with Master: a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
I20260927 10:47:29.453361 24919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:37309
W20260927 10:47:29.551076 25248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:29.551364 25248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:29.551410 25248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:29.556228 25248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:29.556298 25248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:29.556360 25248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:47:29.561789 25248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:40473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=43917
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:29.562829 25248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:29.563880 25248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:29.573810 25256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.573836 25254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.573810 25253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:29.574134 25248 server_base.cc:1034] running on GCE node
I20260927 10:47:29.574740 25248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:29.575431 25248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:29.576622 25248 hybrid_clock.cc:648] HybridClock initialized: now 1790506049576604 us; error 64 us; skew 500 ppm
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:29.579512 25248 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:47:29.580565 25248 webserver.cc:460] Webserver started at http://127.10.96.195:43917/ using document root <none> and password file <none>
I20260927 10:47:29.581399 25248 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:29.581525 25248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:29.585913 25248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:47:29.588083 25263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.589365 25248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:29.589525 25248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "10b8589209b7404fa369524b048fc027"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "3a8d4dc7873aff5da856508cd21c8c6a"
server_key_iv: "e521de4d0fc598b83c9000216145bd2a"
server_key_version: "encryptionkey@0"
I20260927 10:47:29.590075 25248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:29.602347 25248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:29.605624 25248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:29.605859 25248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:29.606494 25248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:29.607391 25248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:29.607474 25248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.607550 25248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:29.607609 25248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.616988 25248 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40473
I20260927 10:47:29.617017 25376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40473 every 8 connection(s)
I20260927 10:47:29.618142 25248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:29.624053 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25248
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:29.633517 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.619972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:38363 (local address 127.10.96.254:33485)
0927 10:47:29.620121 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.620125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.620964 (+ 839us) server_negotiation.cc:407] Connection header received
0927 10:47:29.622473 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.622476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.622524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.622604 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:29.623878 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.624611 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.626071 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.626397 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.626477 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.629406 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:29.629409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:29.629411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:29.629432 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:29.631271 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.631828 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.631830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.631832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.631900 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.632305 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.632310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.632315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.632519 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:29.632614 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.633269 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.633366 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:47:29.633519 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.624744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.10.96.254:33485)
0927 10:47:29.624980 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.624983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.625226 (+ 243us) server_negotiation.cc:407] Connection header received
0927 10:47:29.625377 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.625381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.625429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.625544 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:29.626304 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.627125 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.628079 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.628380 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.628463 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.630159 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:29.630163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:29.630168 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:29.630210 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:29.631665 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.632157 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.632160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.632161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.632225 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:29.632513 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:29.632516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:29.632518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:29.632701 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:29.632848 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.633114 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.633291 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260927 10:47:29.634697 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.620355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:38363)
0927 10:47:29.620801 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.620846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.622251 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.622807 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.622814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.623189 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:29.623700 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.623713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.624778 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.624784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.625948 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.625959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.626549 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.626553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.626788 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.627521 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:29.627538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:29.629213 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:29.631396 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:29.631402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:29.631417 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:29.631695 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:29.631993 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:29.631998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:29.632000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:29.632214 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:29.632632 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:29.632637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:29.632955 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.633202 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.633579 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260927 10:47:29.636098 25377 heartbeater.cc:347] Connected to a master server at 127.10.96.254:33485
I20260927 10:47:29.636312 25377 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:29.636759 25377 heartbeater.cc:511] Master 127.10.96.254:33485 requested a full tablet report, sending...
I20260927 10:47:29.637738 24919 ts_manager.cc:194] Registered new tserver with Master: 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473)
I20260927 10:47:29.638257 24919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:38363
I20260927 10:47:29.647457 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:29.651818 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.648056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.10.96.254:33485)
0927 10:47:29.648176 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.648180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.648338 (+ 158us) server_negotiation.cc:407] Connection header received
0927 10:47:29.648468 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.648470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.648565 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.648696 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:29.648918 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.649439 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.650425 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.650704 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.650789 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.651085 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:29.651475 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:29.651568 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.651630 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.651712 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 10:47:29.652470 24919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:29.656018 24919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:29.664423 25041 tablet_service.cc:1467] Processing CreateTablet for tablet a83a8d55f4494ff4819e4cb35c8c36d9 (DEFAULT_TABLE table=test-table [id=ba84c36264e94e27bf5b6aa487143e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:29.665088 25041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a8d55f4494ff4819e4cb35c8c36d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:29.667871 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.663117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40473 (local address 127.0.0.1:37192)
0927 10:47:29.663257 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.663292 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.663394 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.664162 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.664166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.664182 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:29.664292 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.664299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.665384 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.665390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.666477 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.666486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.667467 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.667471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.667558 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.667580 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.667655 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.667744 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:47:29.667953 24999 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap starting.
I20260927 10:47:29.668524 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.663478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:46087 (local address 127.0.0.1:42570)
0927 10:47:29.663743 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.663760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.663848 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.664492 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.664494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.664512 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:29.664598 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.664606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.665859 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.665862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.666947 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.666955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.668125 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.668128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.668223 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.668245 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.668319 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.668383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260927 10:47:29.668655 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.663672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.10.96.194:46087)
0927 10:47:29.664058 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.664069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.664090 (+ 21us) server_negotiation.cc:407] Connection header received
0927 10:47:29.664169 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.664173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.664366 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.664511 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:29.664688 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.665717 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.667076 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.667894 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.668095 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.668244 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.668348 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.668442 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":87,"threads_started":1}
I20260927 10:47:29.668560 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.663317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.10.96.195:40473)
0927 10:47:29.663652 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.663664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.663689 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:47:29.663788 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.663794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.664024 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.664201 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:29.664412 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.665258 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.666609 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.667339 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.667462 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.667608 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.667692 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.667781 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":116,"threads_started":1}
I20260927 10:47:29.668941 24999 tablet_bootstrap.cc:654] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:29.670063 24999 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: No bootstrap required, opened a new log
I20260927 10:47:29.670135 24999 ts_tablet_manager.cc:1397] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:29.670687 24999 raft_consensus.cc:348] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.670799 24999 raft_consensus.cc:374] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:29.670828 24999 raft_consensus.cc:729] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b2f3fd3ed455da89e9f5045d0692e, State: Initialized, Role: FOLLOWER
I20260927 10:47:29.670994 24999 consensus_queue.cc:260] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.671265 24999 ts_tablet_manager.cc:1428] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 10:47:29.672072 25179 tablet_service.cc:1467] Processing CreateTablet for tablet a83a8d55f4494ff4819e4cb35c8c36d9 (DEFAULT_TABLE table=test-table [id=ba84c36264e94e27bf5b6aa487143e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:29.673295 25179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a8d55f4494ff4819e4cb35c8c36d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:29.675595 25311 tablet_service.cc:1467] Processing CreateTablet for tablet a83a8d55f4494ff4819e4cb35c8c36d9 (DEFAULT_TABLE table=test-table [id=ba84c36264e94e27bf5b6aa487143e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:29.676707 25311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83a8d55f4494ff4819e4cb35c8c36d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:29.681979 25390 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Bootstrap starting.
I20260927 10:47:29.684432 25390 tablet_bootstrap.cc:654] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:29.684604 25391 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Bootstrap starting.
I20260927 10:47:29.685374 25390 log.cc:824] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:29.687222 25391 tablet_bootstrap.cc:654] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:29.687595 25390 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: No bootstrap required, opened a new log
I20260927 10:47:29.687862 25390 ts_tablet_manager.cc:1397] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20260927 10:47:29.688027 25391 log.cc:824] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:29.690059 25391 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: No bootstrap required, opened a new log
I20260927 10:47:29.690248 25391 ts_tablet_manager.cc:1397] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:29.691860 25390 raft_consensus.cc:348] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.692145 25390 raft_consensus.cc:374] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:29.692195 25390 raft_consensus.cc:729] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0996ba87c484dba89b3c6aac854347f, State: Initialized, Role: FOLLOWER
I20260927 10:47:29.692659 25390 consensus_queue.cc:260] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.693363 25245 heartbeater.cc:503] Master 127.10.96.254:33485 was elected leader, sending a full tablet report...
I20260927 10:47:29.693764 25390 ts_tablet_manager.cc:1428] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260927 10:47:29.694051 25391 raft_consensus.cc:348] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.694344 25391 raft_consensus.cc:374] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:29.694399 25391 raft_consensus.cc:729] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b8589209b7404fa369524b048fc027, State: Initialized, Role: FOLLOWER
I20260927 10:47:29.694828 25391 consensus_queue.cc:260] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.695502 25377 heartbeater.cc:503] Master 127.10.96.254:33485 was elected leader, sending a full tablet report...
I20260927 10:47:29.695849 25391 ts_tablet_manager.cc:1428] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260927 10:47:29.717372 25395 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:29.717633 25395 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.718721 25395 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087), 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843)
I20260927 10:47:29.722932 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.719022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:46087 (local address 127.10.96.195:56489)
0927 10:47:29.719144 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.719160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.719254 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.719604 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.719610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.719632 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:29.719744 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.719748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.720697 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.720700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.721737 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.721746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.722509 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.722512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.722627 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.722663 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.722737 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.722813 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":21}
I20260927 10:47:29.722981 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.719140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56489 (local address 127.10.96.194:46087)
0927 10:47:29.719287 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.719293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.719343 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:47:29.719400 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.719405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.719452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.719545 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:29.719897 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.720569 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.721869 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.722391 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.722483 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.722524 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.722765 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.722845 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 10:47:29.723230 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.719456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:59941 (local address 127.10.96.193:35843)
0927 10:47:29.719593 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.719597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.719801 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:47:29.719909 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.719911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.719949 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.720006 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:29.720318 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.720972 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.722142 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.722619 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.722754 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.722798 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.723011 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.723092 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:47:29.723508 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.719306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35843 (local address 127.10.96.195:59941)
0927 10:47:29.719688 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.719702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:29.719796 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:29.720040 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:29.720042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:29.720058 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:29.720180 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.720185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.721092 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.721094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.722019 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:29.722029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.722760 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.722763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:29.722866 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.722888 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:47:29.722965 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 10:47:29.723037 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":2,"thread_start_us":139,"threads_started":1}
I20260927 10:47:29.723654 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a8d55f4494ff4819e4cb35c8c36d9" candidate_uuid: "10b8589209b7404fa369524b048fc027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0996ba87c484dba89b3c6aac854347f" is_pre_election: true
I20260927 10:47:29.724040 25199 raft_consensus.cc:2454] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b8589209b7404fa369524b048fc027 in term 0.
I20260927 10:47:29.724120 25061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a8d55f4494ff4819e4cb35c8c36d9" candidate_uuid: "10b8589209b7404fa369524b048fc027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" is_pre_election: true
I20260927 10:47:29.724447 25061 raft_consensus.cc:2454] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b8589209b7404fa369524b048fc027 in term 0.
I20260927 10:47:29.724591 25264 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f; no voters:
I20260927 10:47:29.724910 25395 raft_consensus.cc:2781] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:29.725008 25395 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:29.725068 25395 raft_consensus.cc:3037] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:29.726289 25395 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.726688 25395 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [CANDIDATE]: Term 1 election: Requested vote from peers a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087), 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843)
I20260927 10:47:29.727151 25199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a8d55f4494ff4819e4cb35c8c36d9" candidate_uuid: "10b8589209b7404fa369524b048fc027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0996ba87c484dba89b3c6aac854347f"
I20260927 10:47:29.727202 25061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83a8d55f4494ff4819e4cb35c8c36d9" candidate_uuid: "10b8589209b7404fa369524b048fc027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
I20260927 10:47:29.727288 25199 raft_consensus.cc:3037] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:29.727310 25061 raft_consensus.cc:3037] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:29.728621 25061 raft_consensus.cc:2454] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b8589209b7404fa369524b048fc027 in term 1.
I20260927 10:47:29.728628 25199 raft_consensus.cc:2454] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b8589209b7404fa369524b048fc027 in term 1.
I20260927 10:47:29.728895 25264 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b8589209b7404fa369524b048fc027, 4e9b2f3fd3ed455da89e9f5045d0692e; no voters:
I20260927 10:47:29.729068 25395 raft_consensus.cc:2781] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:29.729297 25395 raft_consensus.cc:686] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 1 LEADER]: Becoming Leader. State: Replica: 10b8589209b7404fa369524b048fc027, State: Running, Role: LEADER
I20260927 10:47:29.729655 25395 consensus_queue.cc:237] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:29.732888 24918 catalog_manager.cc:5495] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b8589209b7404fa369524b048fc027 (127.10.96.195). New cstate: current_term: 1 leader_uuid: "10b8589209b7404fa369524b048fc027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } health_report { overall_health: UNKNOWN } } }
W20260927 10:47:29.734549 25108 tablet.cc:2369] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:29.750994 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.747450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.10.96.195:40473)
0927 10:47:29.747584 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:29.747588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:29.747602 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:47:29.747730 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:29.747732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:29.747787 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:29.747880 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:29.748074 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.748624 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.749454 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:29.749762 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:29.749851 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:29.750116 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:29.750550 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:29.750647 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:29.750704 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 10:47:29.750763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:47:29.757210 25199 raft_consensus.cc:1261] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Refusing update from remote peer 10b8589209b7404fa369524b048fc027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:29.757239 25061 raft_consensus.cc:1261] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Refusing update from remote peer 10b8589209b7404fa369524b048fc027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:29.757979 25397 consensus_queue.cc:1035] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:29.758175 25395 consensus_queue.cc:1035] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:29.764353 25400 mvcc.cc:205] Tried to move back new op lower bound from 7333912779797872640 to 7333912779695325184. Current Snapshot: MvccSnapshot[applied={T|T < 7333912779797872640}]
I20260927 10:47:29.764467 25401 mvcc.cc:205] Tried to move back new op lower bound from 7333912779797872640 to 7333912779695325184. Current Snapshot: MvccSnapshot[applied={T|T < 7333912779797872640}]
I20260927 10:47:29.767383 25402 mvcc.cc:205] Tried to move back new op lower bound from 7333912779797872640 to 7333912779695325184. Current Snapshot: MvccSnapshot[applied={T|T < 7333912779797872640}]
I20260927 10:47:29.772948 24919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:29.773125 24919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:29.775518 24919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Sending DeleteTablet for 3 replicas of tablet a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:29.776216 25311 tablet_service.cc:1514] Processing DeleteTablet for tablet a83a8d55f4494ff4819e4cb35c8c36d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:29 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:37192
I20260927 10:47:29.776298 25179 tablet_service.cc:1514] Processing DeleteTablet for tablet a83a8d55f4494ff4819e4cb35c8c36d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:29 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:42570
I20260927 10:47:29.776363 25041 tablet_service.cc:1514] Processing DeleteTablet for tablet a83a8d55f4494ff4819e4cb35c8c36d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:29 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:48720
I20260927 10:47:29.776688 25116 tablet_replica.cc:330] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: stopping tablet replica
I20260927 10:47:29.776830 25116 raft_consensus.cc:2229] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:29.776911 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24957
I20260927 10:47:29.776984 25116 raft_consensus.cc:2258] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:29.777516 25410 tablet_replica.cc:330] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: stopping tablet replica
I20260927 10:47:29.776933 25411 tablet_replica.cc:330] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: stopping tablet replica
I20260927 10:47:29.778256 25410 raft_consensus.cc:2229] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:29.777897 25411 raft_consensus.cc:2229] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:29.778643 25410 raft_consensus.cc:2258] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:29.779155 25411 raft_consensus.cc:2258] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:29.780725 25411 ts_tablet_manager.cc:1905] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:29.781039 25410 ts_tablet_manager.cc:1905] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:29.783891 25410 ts_tablet_manager.cc:1918] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:29.784020 25410 log.cc:1196] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:29.784395 25410 ts_tablet_manager.cc:1939] T a83a8d55f4494ff4819e4cb35c8c36d9 P a0996ba87c484dba89b3c6aac854347f: Deleting consensus metadata
I20260927 10:47:29.784535 25411 ts_tablet_manager.cc:1918] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:29.784636 25411 log.cc:1196] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:29.784963 25411 ts_tablet_manager.cc:1939] T a83a8d55f4494ff4819e4cb35c8c36d9 P 10b8589209b7404fa369524b048fc027: Deleting consensus metadata
W20260927 10:47:29.785499 24903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:35843 (error 108)
I20260927 10:47:29.786090 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 25111
I20260927 10:47:29.786139 24906 catalog_manager.cc:4841] TS 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): tablet a83a8d55f4494ff4819e4cb35c8c36d9 (table test-table [id=ba84c36264e94e27bf5b6aa487143e48]) successfully deleted
I20260927 10:47:29.790431 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.790055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:35843 (local address 127.0.0.1:48734)
0927 10:47:29.790233 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.790322 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:35843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 10:47:29.790622 24903 catalog_manager.cc:4584] TS 4e9b2f3fd3ed455da89e9f5045d0692e (127.10.96.193:35843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a83a8d55f4494ff4819e4cb35c8c36d9: Network error: Client connection negotiation failed: client connection to 127.10.96.193:35843: connect: Connection refused (error 111)
I20260927 10:47:29.792152 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 25248
I20260927 10:47:29.793776 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:29.792309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:46087 (local address 127.0.0.1:42574)
0927 10:47:29.793588 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:29.793692 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:46087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1127}
I20260927 10:47:29.797977 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 24886
I20260927 10:47:29.804600 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33485
--webserver_interface=127.10.96.254
--webserver_port=45051
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:29.916553 25413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:29.916896 25413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:29.916939 25413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:29.921030 25413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:29.921095 25413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:29.921114 25413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:29.921132 25413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:29.921149 25413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:29.926559 25413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=45051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:29.927676 25413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:29.928745 25413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:47:29.937388 25413 server_base.cc:1034] running on GCE node
W20260927 10:47:29.937314 25421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.937314 25419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:29.937315 25418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:29.938139 25413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:29.939318 25413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:29.940519 25413 hybrid_clock.cc:648] HybridClock initialized: now 1790506049940506 us; error 33 us; skew 500 ppm
Sep 27 10:47:29 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506049, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:29.943403 25413 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:29.944344 25413 webserver.cc:460] Webserver started at http://127.10.96.254:45051/ using document root <none> and password file <none>
I20260927 10:47:29.944916 25413 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:29.945007 25413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:29.948915 25413 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:29.951267 25428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:29.952709 25413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:47:29.952876 25413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "2e7e801fed454d07a7932227a6abf5bf"
format_stamp: "Formatted at 2026-09-27 10:47:27 on dist-test-slave-wmg0"
server_key: "bc8a9ec185889635069e926109279942"
server_key_iv: "f2bfdcbf5368ce2b6a116c3856d7fea2"
server_key_version: "encryptionkey@0"
I20260927 10:47:29.953464 25413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:29.972411 25413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:29.975931 25413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:29.976146 25413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:29.983762 25413 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:33485
I20260927 10:47:29.984046 25480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:33485 every 8 connection(s)
I20260927 10:47:29.984891 25413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:29.989089 25481 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:47:29.990765 25481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap starting.
I20260927 10:47:29.991333 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25413
I20260927 10:47:29.991809 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:35843
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:33485
--webserver_port=40051
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:30.009325 25481 log.cc:824] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:30.020493 25481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:30.020888 25481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf: Bootstrap complete.
I20260927 10:47:30.024863 25481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:30.025238 25481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7e801fed454d07a7932227a6abf5bf, State: Initialized, Role: FOLLOWER
I20260927 10:47:30.025763 25481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:30.025880 25481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:30.025969 25481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:30.026064 25481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:47:30.027894 25481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:30.028227 25481 leader_election.cc:302] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e7e801fed454d07a7932227a6abf5bf; no voters:
I20260927 10:47:30.028529 25481 leader_election.cc:288] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:47:30.028661 25485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:47:30.028898 25485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [term 3 LEADER]: Becoming Leader. State: Replica: 2e7e801fed454d07a7932227a6abf5bf, State: Running, Role: LEADER
I20260927 10:47:30.029212 25485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } }
I20260927 10:47:30.029796 25481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:30.030726 25486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2e7e801fed454d07a7932227a6abf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } } }
I20260927 10:47:30.030890 25486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:30.030768 25487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e7e801fed454d07a7932227a6abf5bf. Latest consensus state: current_term: 3 leader_uuid: "2e7e801fed454d07a7932227a6abf5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e7e801fed454d07a7932227a6abf5bf" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33485 } } }
I20260927 10:47:30.031122 25487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e7e801fed454d07a7932227a6abf5bf [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:30.031275 25494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:30.036653 25494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:30.037425 25494 catalog_manager.cc:1259] Loaded cluster ID: 92a7f7cce9824ca0bac7354ff0c6ca3e
I20260927 10:47:30.037508 25494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 10:47:30.038528 25494 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe2158e39ad google::LogMessage::Fail() at ??:0
@ 0x7fe2158e688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe2158e34de google::LogMessage::Flush() at ??:0
@ 0x7fe2158e4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe220222fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe220223d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe220220579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe220264640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe217b353d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe217b35c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe217b373f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe217b26512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe21916a6db start_thread at ??:0
@ 0x7fe2143e161f clone at ??:0
W20260927 10:47:30.123939 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:30.124226 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:30.124315 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:30.128475 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:30.128557 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:30.128676 25483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:47:30.134296 25483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:35843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=40051
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:30.135411 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:30.136467 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:30.147396 25523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:30.147411 25522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:30.147460 25525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:30.147945 25483 server_base.cc:1034] running on GCE node
I20260927 10:47:30.148394 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:30.149104 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:30.150286 25483 hybrid_clock.cc:648] HybridClock initialized: now 1790506050150262 us; error 47 us; skew 500 ppm
Sep 27 10:47:30 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:30.153296 25483 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:47:30.154436 25483 webserver.cc:460] Webserver started at http://127.10.96.193:40051/ using document root <none> and password file <none>
I20260927 10:47:30.155054 25483 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:30.155146 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:30.159351 25483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:30.161677 25532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.163036 25483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:30.163177 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "4e9b2f3fd3ed455da89e9f5045d0692e"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "0a5f83790299cf319c13d74f6b14b526"
server_key_iv: "1fef37ff708abb40e02e81d35e172443"
server_key_version: "encryptionkey@0"
I20260927 10:47:30.163755 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:30.185029 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:30.188733 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:30.188935 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:30.189572 25483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:30.190706 25539 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:47:30.194837 25483 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:47:30.194898 25483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 10:47:30.194945 25483 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:47:30.196626 25483 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:47:30.196688 25483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:47:30.196992 25539 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap starting.
I20260927 10:47:30.207370 25483 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:35843
I20260927 10:47:30.207430 25646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:35843 every 8 connection(s)
I20260927 10:47:30.208640 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:30.208870 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25483
I20260927 10:47:30.209321 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:46087
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:33485
--webserver_port=42829
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:30.219707 25539 log.cc:824] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:30.229357 25539 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:47:30.229903 25539 tablet_bootstrap.cc:492] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Bootstrap complete.
I20260927 10:47:30.231662 25539 ts_tablet_manager.cc:1397] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.009s
I20260927 10:47:30.234321 25539 raft_consensus.cc:348] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:30.234836 25539 raft_consensus.cc:729] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b2f3fd3ed455da89e9f5045d0692e, State: Initialized, Role: FOLLOWER
I20260927 10:47:30.235384 25539 consensus_queue.cc:260] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:30.236372 25539 ts_tablet_manager.cc:1428] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 10:47:30.283289 25650 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:47:30.211920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:51127)
0927 10:47:30.217839 (+ 5919us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.217916 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:30.219101 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:30.282311 (+ 63210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5639,"thread_start_us":149,"threads_started":1}
W20260927 10:47:30.283936 25647 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:33485 (0 consecutive failures): Network error: Failed to ping master at 127.10.96.254:33485: Client connection negotiation failed: client connection to 127.10.96.254:33485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:47:30.284763 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.284445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:52861)
0927 10:47:30.284627 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.284700 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 10:47:30.285538 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.285266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:36237)
0927 10:47:30.285381 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.285427 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:47:30.285797 25647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:47:30.342461 25649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:30.342871 25649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:30.342998 25649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:30.347353 25649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:30.347416 25649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:30.347478 25649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:47:30.352710 25649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:46087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=42829
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:30.353780 25649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:30.354777 25649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:30.364363 25657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:30.364358 25660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:30.364373 25658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:30.364636 25649 server_base.cc:1034] running on GCE node
I20260927 10:47:30.365238 25649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:30.365911 25649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:30.367100 25649 hybrid_clock.cc:648] HybridClock initialized: now 1790506050367083 us; error 32 us; skew 500 ppm
Sep 27 10:47:30 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:30.370332 25649 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:47:30.371362 25649 webserver.cc:460] Webserver started at http://127.10.96.194:42829/ using document root <none> and password file <none>
I20260927 10:47:30.371956 25649 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:30.372047 25649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:30.376113 25649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:30.378371 25667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.379637 25649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:30.379812 25649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a0996ba87c484dba89b3c6aac854347f"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "8125e3e3115874280bb09bb143c2241e"
server_key_iv: "133716ce437b3f9e2339764ad243f2c9"
server_key_version: "encryptionkey@0"
I20260927 10:47:30.380374 25649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:30.420674 25649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:30.424028 25649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:30.424245 25649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:30.424885 25649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:30.425761 25649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:30.425832 25649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.425896 25649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:30.425940 25649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.435230 25649 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:46087
I20260927 10:47:30.435267 25780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:46087 every 8 connection(s)
I20260927 10:47:30.436251 25649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:47:30.438886 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25649
I20260927 10:47:30.439355 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:40473
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:33485
--webserver_port=43917
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:47:30.440828 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.438639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:33323)
0927 10:47:30.439233 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.439823 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":111,"threads_started":1}
W20260927 10:47:30.441298 25781 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:33485 (0 consecutive failures): Network error: Failed to ping master at 127.10.96.254:33485: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
I20260927 10:47:30.442013 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.441798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:43153)
0927 10:47:30.441909 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.441958 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:47:30.442718 25783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.442486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:54421)
0927 10:47:30.442593 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.442642 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:47:30.442929 25781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:47:30.551057 25784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:30.551330 25784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:30.551374 25784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:30.555521 25784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:30.555584 25784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:30.555677 25784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:47:30.561226 25784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:40473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=43917
--tserver_master_addrs=127.10.96.254:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:30.562300 25784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:30.563325 25784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:30.572964 25789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:30.572945 25792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:30.572947 25790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:30.573256 25784 server_base.cc:1034] running on GCE node
I20260927 10:47:30.573728 25784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:30.574366 25784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:30.575554 25784 hybrid_clock.cc:648] HybridClock initialized: now 1790506050575514 us; error 50 us; skew 500 ppm
Sep 27 10:47:30 dist-test-slave-wmg0 krb5kdc[24339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:30.578327 25784 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:47:30.579291 25784 webserver.cc:460] Webserver started at http://127.10.96.195:43917/ using document root <none> and password file <none>
I20260927 10:47:30.579864 25784 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:30.579958 25784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:30.584107 25784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:47:30.587380 25799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.588704 25784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:47:30.588814 25784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "10b8589209b7404fa369524b048fc027"
format_stamp: "Formatted at 2026-09-27 10:47:28 on dist-test-slave-wmg0"
server_key: "3a8d4dc7873aff5da856508cd21c8c6a"
server_key_iv: "e521de4d0fc598b83c9000216145bd2a"
server_key_version: "encryptionkey@0"
I20260927 10:47:30.589401 25784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:30.610589 25784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:30.613948 25784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:30.614161 25784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:30.614812 25784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:30.615670 25784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:30.615720 25784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.615769 25784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:30.615785 25784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:30.625208 25784 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40473
I20260927 10:47:30.625240 25912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40473 every 8 connection(s)
I20260927 10:47:30.626282 25784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:30.626863 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25784
I20260927 10:47:30.630815 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.629068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:38749)
0927 10:47:30.629637 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.630076 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
W20260927 10:47:30.631260 25913 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:33485 (0 consecutive failures): Network error: Failed to ping master at 127.10.96.254:33485: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
I20260927 10:47:30.631875 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.631658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:43949)
0927 10:47:30.631767 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.631823 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 10:47:30.632534 25916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:30.632318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:34589)
0927 10:47:30.632427 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:30.632474 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:47:30.632721 25913 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:47:31.286938 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.286448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:38589)
0927 10:47:31.286745 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.286809 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
I20260927 10:47:31.381417 25918 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:31.381769 25918 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:31.382858 25918 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
Sep 27 10:47:31 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.194@KRBTEST.COM
Sep 27 10:47:31 dist-test-slave-wmg0 krb5kdc[24339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506050, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.195@KRBTEST.COM
I20260927 10:47:31.398268 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.383407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:48447 (local address 127.10.96.195:40473)
0927 10:47:31.383722 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:31.383738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:31.384022 (+ 284us) server_negotiation.cc:407] Connection header received
0927 10:47:31.384863 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:31.384870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:31.385168 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:31.385518 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:31.387228 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.388333 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.389577 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.390023 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.390202 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:31.393242 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:31.393247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:31.393258 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:31.393285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:31.395216 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:31.395736 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:31.395741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:31.395745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:31.395876 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:31.396304 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:31.396307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:31.396309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:31.396873 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:31.397011 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:31.397800 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 10:47:31.398076 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260927 10:47:31.398294 25917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.383112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:46087 (local address 127.10.96.193:58041)
0927 10:47:31.383373 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.383390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:31.383535 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:31.385625 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:31.385631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:31.386238 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:31.387041 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:31.387061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.388475 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.388478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:31.389393 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:31.389403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.390174 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.390177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:31.390382 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:31.391111 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:31.391123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:31.392755 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:31.395123 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:31.395130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:31.395143 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:31.395566 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:31.395981 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:31.395984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:31.395986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:31.396198 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:31.397001 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:31.397009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:31.397408 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 10:47:31.397663 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 10:47:31.397986 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":146,"spinlock_wait_cycles":45696}
I20260927 10:47:31.398869 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.383109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40473 (local address 127.10.96.193:48447)
0927 10:47:31.383920 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.383932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:31.384024 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:31.385629 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:31.385632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:31.386238 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:31.387041 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:31.387061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.388514 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.388517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:31.389392 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:31.389401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.390143 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.390147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:31.390384 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:31.391046 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:31.391062 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:31.393060 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:31.395319 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:31.395322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:31.395324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:31.395633 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:31.395981 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:31.395984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:31.395986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:31.396198 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:31.397001 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:31.397009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:31.397408 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 10:47:31.397758 (+ 350us) client_negotiation.cc:240] Negotiation successful
0927 10:47:31.398759 (+ 1001us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":1754,"spinlock_wait_cycles":24576,"thread_start_us":90,"threads_started":1}
I20260927 10:47:31.399078 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.383407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:58041 (local address 127.10.96.194:46087)
0927 10:47:31.383778 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:31.383787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:31.383828 (+ 41us) server_negotiation.cc:407] Connection header received
0927 10:47:31.384679 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:31.384687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:31.385169 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:31.385518 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:31.387269 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.388329 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.389576 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:31.390020 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:31.390203 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:31.393011 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:31.393019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:31.393035 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:31.393067 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:31.394994 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:31.395714 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:31.395720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:31.395726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:31.395876 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:31.396304 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:31.396307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:31.396309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:31.396873 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:31.397012 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:31.398599 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0927 10:47:31.398870 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
W20260927 10:47:31.401172 25536 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
W20260927 10:47:31.401358 25533 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:31.401500 25533 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9b2f3fd3ed455da89e9f5045d0692e; no voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f
I20260927 10:47:31.401816 25918 raft_consensus.cc:2726] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:47:31.443826 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.443445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:43355)
0927 10:47:31.443682 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.443733 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260927 10:47:31.633852 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:31.633316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:51763)
0927 10:47:31.633691 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:31.633750 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":73,"threads_started":1}
I20260927 10:47:32.288187 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:32.287769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:42157)
0927 10:47:32.288013 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:32.288073 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260927 10:47:32.445066 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:32.444620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:51075)
0927 10:47:32.444888 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:32.444947 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260927 10:47:32.635067 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:32.634623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:43811)
0927 10:47:32.634891 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:32.634955 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260927 10:47:32.988721 25927 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:32.988847 25927 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:32.989270 25927 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
W20260927 10:47:32.990129 25536 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
W20260927 10:47:32.990242 25533 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:32.990329 25533 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9b2f3fd3ed455da89e9f5045d0692e; no voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f
I20260927 10:47:32.990461 25927 raft_consensus.cc:2726] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:47:33.289563 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:33.289044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:53637)
0927 10:47:33.289334 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:33.289395 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":90,"threads_started":1}
I20260927 10:47:33.446381 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:33.445961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:34093)
0927 10:47:33.446220 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:33.446291 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260927 10:47:33.636224 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:33.635788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:34983)
0927 10:47:33.636056 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:33.636110 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":74,"threads_started":1}
I20260927 10:47:34.290815 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:34.290388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:39805)
0927 10:47:34.290646 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:34.290706 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260927 10:47:34.447762 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:34.447229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:54107)
0927 10:47:34.447564 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:34.447664 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260927 10:47:34.540146 25933 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:34.540303 25933 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:34.540762 25933 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
W20260927 10:47:34.541600 25536 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
W20260927 10:47:34.541708 25533 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:34.541796 25536 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9b2f3fd3ed455da89e9f5045d0692e; no voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f
I20260927 10:47:34.541990 25933 raft_consensus.cc:2726] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:47:34.637290 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:34.636910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:59551)
0927 10:47:34.637136 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:34.637202 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260927 10:47:35.292341 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:35.291810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:52589)
0927 10:47:35.292182 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:35.292244 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":92,"threads_started":1}
I20260927 10:47:35.448942 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:35.448493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:41507)
0927 10:47:35.448795 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:35.448853 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
I20260927 10:47:35.638576 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:35.638139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:47897)
0927 10:47:35.638420 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:35.638485 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260927 10:47:36.293823 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:36.293262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:45435)
0927 10:47:36.293621 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:36.293689 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":72,"threads_started":1}
I20260927 10:47:36.450344 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:36.449924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:49843)
0927 10:47:36.450192 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:36.450250 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260927 10:47:36.639731 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:36.639275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:39955)
0927 10:47:36.639586 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:36.639639 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260927 10:47:37.295022 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:37.294588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:39747)
0927 10:47:37.294862 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:37.294926 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260927 10:47:37.451594 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:37.451153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:56575)
0927 10:47:37.451446 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:37.451501 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260927 10:47:37.640910 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:37.640459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:45011)
0927 10:47:37.640763 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:37.640818 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
I20260927 10:47:38.207342 25944 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:38.207477 25944 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:38.207986 25944 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
W20260927 10:47:38.208781 25536 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
W20260927 10:47:38.208878 25533 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:38.208976 25533 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9b2f3fd3ed455da89e9f5045d0692e; no voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f
I20260927 10:47:38.209142 25944 raft_consensus.cc:2726] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:47:38.296391 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:38.295919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:44545)
0927 10:47:38.296224 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:38.296294 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260927 10:47:38.452801 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:38.452333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:43973)
0927 10:47:38.452635 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:38.452697 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":86,"threads_started":1}
I20260927 10:47:38.642200 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:38.641763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:57903)
0927 10:47:38.642046 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:38.642107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
I20260927 10:47:39.297706 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.297201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:59423)
0927 10:47:39.297519 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.297578 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260927 10:47:39.454126 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.453675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:48529)
0927 10:47:39.453913 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.453979 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260927 10:47:39.643409 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:39.642967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:44595)
0927 10:47:39.643238 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:39.643297 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260927 10:47:40.298947 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.298538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:56641)
0927 10:47:40.298791 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.298851 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260927 10:47:40.455396 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.454972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:41683)
0927 10:47:40.455236 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.455303 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260927 10:47:40.644785 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:40.644310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:47447)
0927 10:47:40.644625 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:40.644687 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260927 10:47:41.300392 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:41.299920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:42049)
0927 10:47:41.300228 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:41.300293 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
I20260927 10:47:41.456655 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:41.456155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:56981)
0927 10:47:41.456452 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:41.456553 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260927 10:47:41.646081 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:41.645692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:57345)
0927 10:47:41.645937 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:41.645989 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260927 10:47:42.301730 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:44305)
0927 10:47:42.301542 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:42.301631 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
I20260927 10:47:42.457939 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.457548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:48739)
0927 10:47:42.457794 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:42.457846 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260927 10:47:42.647233 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:42.646836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:56143)
0927 10:47:42.647086 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:42.647140 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260927 10:47:43.302932 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.302501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:52321)
0927 10:47:43.302778 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.302835 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
I20260927 10:47:43.459263 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.458685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:50801)
0927 10:47:43.459040 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.459119 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":81,"threads_started":1}
I20260927 10:47:43.493240 25962 raft_consensus.cc:482] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:43.493374 25962 raft_consensus.cc:504] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b8589209b7404fa369524b048fc027" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40473 } } peers { permanent_uuid: "a0996ba87c484dba89b3c6aac854347f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 46087 } } peers { permanent_uuid: "4e9b2f3fd3ed455da89e9f5045d0692e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 35843 } }
I20260927 10:47:43.493906 25962 leader_election.cc:288] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473), a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087)
W20260927 10:47:43.494613 25533 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0996ba87c484dba89b3c6aac854347f (127.10.96.194:46087): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
W20260927 10:47:43.494721 25536 leader_election.cc:341] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10b8589209b7404fa369524b048fc027 (127.10.96.195:40473): Not found: Tablet not found: a83a8d55f4494ff4819e4cb35c8c36d9
I20260927 10:47:43.494839 25536 leader_election.cc:302] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9b2f3fd3ed455da89e9f5045d0692e; no voters: 10b8589209b7404fa369524b048fc027, a0996ba87c484dba89b3c6aac854347f
I20260927 10:47:43.494997 25962 raft_consensus.cc:2726] T a83a8d55f4494ff4819e4cb35c8c36d9 P 4e9b2f3fd3ed455da89e9f5045d0692e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 10:47:43.648516 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:43.647993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:55057)
0927 10:47:43.648299 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:43.648369 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260927 10:47:44.304193 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.303739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:49525)
0927 10:47:44.304030 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.304093 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260927 10:47:44.460595 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.460102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:42963)
0927 10:47:44.460395 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.460475 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260927 10:47:44.649806 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:44.649276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.195:41403)
0927 10:47:44.649624 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:44.649704 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":73,"threads_started":1}
I20260927 10:47:45.305722 25967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:45.305190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.193:49215)
0927 10:47:45.305549 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:45.305621 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260927 10:47:45.461968 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:45.461569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:33485 (local address 127.10.96.194:50215)
0927 10:47:45.461814 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:45.461875 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 10:47:45.628902 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 25483
I20260927 10:47:45.635036 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 25649
I20260927 10:47:45.640426 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 25784
2026-09-27T10:47:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19166 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:47:45 dist-test-slave-wmg0 krb5kdc[25971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:47:45 dist-test-slave-wmg0 krb5kdc[25971](info): set up 2 sockets
Sep 27 10:47:45 dist-test-slave-wmg0 krb5kdc[25971](info): commencing operation
krb5kdc: starting...
W20260927 10:47:46.681633 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:47:46 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:47:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:47:46.804564 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43695
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:46.912807 25984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:46.913070 25984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:46.913110 25984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:46.917124 25984 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:47:46.917179 25984 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:47:46.917196 25984 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:47:46.917209 25984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:47:46.917227 25984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:46.917280 25984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:46.917318 25984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:47:46.917338 25984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:47:46.922566 25984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:46.923658 25984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:46.924697 25984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:46.933377 25989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.933444 25984 server_base.cc:1034] running on GCE node
W20260927 10:47:46.933665 25990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:46.933516 25992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:46.934304 25984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:46.935420 25984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:46.936623 25984 hybrid_clock.cc:648] HybridClock initialized: now 1790506066936611 us; error 33 us; skew 500 ppm
Sep 27 10:47:46 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:46.943408 25984 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:47:46.944254 25984 webserver.cc:460] Webserver started at http://127.10.96.254:36889/ using document root <none> and password file <none>
I20260927 10:47:46.944794 25984 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:46.944851 25984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:46.945020 25984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:46.947201 25984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "6d6fe792107e433f9f0abcda363c876b"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-wmg0"
server_key: "3261ea40033a140f4500e04b8bdcb31c"
server_key_iv: "cbd6370e5bcd0f429ee3c2b8f02d0bda"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.947705 25984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "6d6fe792107e433f9f0abcda363c876b"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-wmg0"
server_key: "3261ea40033a140f4500e04b8bdcb31c"
server_key_iv: "cbd6370e5bcd0f429ee3c2b8f02d0bda"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.951375 25984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 10:47:46.953744 25999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:46.955049 25984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:47:46.955145 25984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "6d6fe792107e433f9f0abcda363c876b"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-wmg0"
server_key: "3261ea40033a140f4500e04b8bdcb31c"
server_key_iv: "cbd6370e5bcd0f429ee3c2b8f02d0bda"
server_key_version: "encryptionkey@0"
I20260927 10:47:46.955226 25984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:46.978441 25984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:46.981675 25984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:46.981829 25984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:46.988880 25984 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43695
I20260927 10:47:46.989229 26051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43695 every 8 connection(s)
I20260927 10:47:46.989921 25984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:47:46.991062 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 25984
I20260927 10:47:46.991173 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:47:46.991460 10627 external_mini_cluster.cc:1428] Setting key 184bc06a29103e256f2aca61a1f69936
I20260927 10:47:46.993314 26052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.001029 26052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap starting.
Sep 27 10:47:46 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:47.003911 26052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.004801 26052 log.cc:824] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.006886 26052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: No bootstrap required, opened a new log
I20260927 10:47:47.006918 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:46.992897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.10.96.254:43695)
0927 10:47:46.993348 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:46.993370 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:46.993413 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:47:46.994349 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:46.994373 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:46.994802 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:46.995205 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:46.995858 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.997495 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.998396 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:46.999091 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:46.999265 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.001958 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.001965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.001981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:47.002012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.004052 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.004477 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.004483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.004489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.004633 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.004896 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.004899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.004901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.005404 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.005672 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.005928 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.006230 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260927 10:47:47.010702 26052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:47:47.010941 26052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.011029 26052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6fe792107e433f9f0abcda363c876b, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.011473 26052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:47:47.011653 26052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:47:47.011732 26052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:47:47.011830 26052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.012777 26052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:47:47.013113 26052 leader_election.cc:302] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6fe792107e433f9f0abcda363c876b; no voters:
I20260927 10:47:47.013422 26052 leader_election.cc:288] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:47:47.013650 26057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:47.013939 26057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 LEADER]: Becoming Leader. State: Replica: 6d6fe792107e433f9f0abcda363c876b, State: Running, Role: LEADER
I20260927 10:47:47.014253 26057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:47:47.014662 26052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:47:47.015880 26059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d6fe792107e433f9f0abcda363c876b. Latest consensus state: current_term: 1 leader_uuid: "6d6fe792107e433f9f0abcda363c876b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } } }
I20260927 10:47:47.016014 26059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: This master's current role is: LEADER
I20260927 10:47:47.016196 26058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d6fe792107e433f9f0abcda363c876b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } } }
I20260927 10:47:47.016403 26058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: This master's current role is: LEADER
W20260927 10:47:47.019586 26073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:47:47.019660 26073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:47:47.019739 26069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:47:47.020539 26069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:47:47.025579 26069 catalog_manager.cc:1347] Generated new cluster ID: 1428d134ddfe4db885055ebed40fa13b
I20260927 10:47:47.025640 26069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:47:47.038262 26069 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:47:47.039294 26069 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:47:47.046937 26069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Generated new TSK 0
I20260927 10:47:47.047747 26069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:47:47.084940 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43695
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:47.196995 26078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:47.197258 26078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:47.197302 26078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:47.201560 26078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:47.201622 26078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:47.201681 26078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:47:47.206893 26078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:47.207906 26078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:47.208904 26078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:47.218415 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.218438 26084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.218417 26083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:47.218730 26078 server_base.cc:1034] running on GCE node
I20260927 10:47:47.219122 26078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:47.219710 26078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:47.220867 26078 hybrid_clock.cc:648] HybridClock initialized: now 1790506067220816 us; error 61 us; skew 500 ppm
Sep 27 10:47:47 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:47.223884 26078 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:47:47.224740 26078 webserver.cc:460] Webserver started at http://127.10.96.193:40431/ using document root <none> and password file <none>
I20260927 10:47:47.225294 26078 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:47.225353 26078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:47.225597 26078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:47.227718 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "cf251b4b08fc4ce59de7feae7a00859a"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "d5c9e4b3e48325ea8aea40733f9d2522"
server_key_iv: "7eab34e6dd5b3dfd49752ad1a03bb204"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.228281 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "cf251b4b08fc4ce59de7feae7a00859a"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "d5c9e4b3e48325ea8aea40733f9d2522"
server_key_iv: "7eab34e6dd5b3dfd49752ad1a03bb204"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.232228 26078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:47:47.234831 26093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.236099 26078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:47:47.236244 26078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "cf251b4b08fc4ce59de7feae7a00859a"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "d5c9e4b3e48325ea8aea40733f9d2522"
server_key_iv: "7eab34e6dd5b3dfd49752ad1a03bb204"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.236363 26078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.249306 26078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.252703 26078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.252908 26078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.253877 26078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:47.254787 26078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:47.254861 26078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.254931 26078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:47.254972 26078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.265143 26078 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:44649
I20260927 10:47:47.265183 26206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:44649 every 8 connection(s)
I20260927 10:47:47.266801 26078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:47:47.272193 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26078
I20260927 10:47:47.272334 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:47:47.272660 10627 external_mini_cluster.cc:1428] Setting key ffe3ce99cea90fc0a0c06a5915b70f08
Sep 27 10:47:47 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:47:47.284231 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.269508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:36319 (local address 127.10.96.254:43695)
0927 10:47:47.269658 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.269666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.271031 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 10:47:47.272802 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.272806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.272876 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.272952 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.274310 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.274957 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.276502 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.276894 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.277014 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.279916 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.279920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.279922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:47.279948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.281774 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.282475 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.282478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.282480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.282577 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.283035 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.283038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.283040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.283239 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.283351 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.283892 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.284031 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:47:47.285144 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.270176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:36319)
0927 10:47:47.270855 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.270919 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.272489 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.273173 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.273180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.273617 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:47.274151 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.274171 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.275105 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.275112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.276342 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.276352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.277060 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.277063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.277213 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.277939 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:47.277956 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:47.279696 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:47.281897 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:47.281903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:47.281920 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:47.282316 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:47.282728 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:47.282731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:47.282733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:47.282897 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:47.283362 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:47.283371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:47.283645 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.283839 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.284206 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":203,"threads_started":1}
I20260927 10:47:47.286595 26207 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43695
I20260927 10:47:47.286875 26207 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:47.287376 26207 heartbeater.cc:511] Master 127.10.96.254:43695 requested a full tablet report, sending...
I20260927 10:47:47.288983 26016 ts_manager.cc:194] Registered new tserver with Master: cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:47:47.290290 26016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:36319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:47:47.302048 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43695
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:47.413192 26212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:47.413501 26212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:47.413574 26212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:47.417950 26212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:47.418022 26212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:47.418084 26212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:47:47.423594 26212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:47.424649 26212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:47.425694 26212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:47.435493 26217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.435565 26218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.435549 26220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:47.435642 26212 server_base.cc:1034] running on GCE node
I20260927 10:47:47.436203 26212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:47.436825 26212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:47.437981 26212 hybrid_clock.cc:648] HybridClock initialized: now 1790506067437967 us; error 28 us; skew 500 ppm
Sep 27 10:47:47 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:47.440886 26212 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:47:47.441877 26212 webserver.cc:460] Webserver started at http://127.10.96.194:39613/ using document root <none> and password file <none>
I20260927 10:47:47.442493 26212 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:47.442561 26212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:47.442926 26212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:47.444973 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "e4c70b545b5f4664a0e16e7a950dd586"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "5ba267274247c06317d2b9ab7ad8bdba"
server_key_iv: "f5a6eb2d63e07d5b125175550539c66f"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.445653 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "e4c70b545b5f4664a0e16e7a950dd586"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "5ba267274247c06317d2b9ab7ad8bdba"
server_key_iv: "f5a6eb2d63e07d5b125175550539c66f"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.449615 26212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 10:47:47.451983 26227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.453286 26212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:47.453408 26212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "e4c70b545b5f4664a0e16e7a950dd586"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "5ba267274247c06317d2b9ab7ad8bdba"
server_key_iv: "f5a6eb2d63e07d5b125175550539c66f"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.453572 26212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.474793 26212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.477741 26212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.477934 26212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.478540 26212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:47.479319 26212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:47.479363 26212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.479427 26212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:47.479465 26212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.488313 26212 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:39237
I20260927 10:47:47.488336 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:39237 every 8 connection(s)
I20260927 10:47:47.489229 26212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
Sep 27 10:47:47 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:47.498872 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26212
I20260927 10:47:47.499019 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:47:47.499320 10627 external_mini_cluster.cc:1428] Setting key 71884d0d686dea493df8938150f29790
I20260927 10:47:47.503914 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.491145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:41209 (local address 127.10.96.254:43695)
0927 10:47:47.491305 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.491309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.492101 (+ 792us) server_negotiation.cc:407] Connection header received
0927 10:47:47.493403 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.493407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.493489 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.493604 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.494804 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.495336 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.496484 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.496723 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.496839 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.499430 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.499434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.499436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:47.499465 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.501667 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.502164 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.502170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.502175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.502268 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.502663 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.502668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.502672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.502932 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.503011 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.503692 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.503795 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 10:47:47.504840 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.491495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:41209)
0927 10:47:47.491937 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.491983 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.493194 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.493800 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.493808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.494178 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:47.494654 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.494667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.495466 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.495476 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.496356 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.496365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.496889 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.496893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.497065 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.497719 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:47.497734 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:47.499260 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:47.501784 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:47.501789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:47.501802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:47.502063 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:47.502384 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:47.502387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:47.502389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:47.502558 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:47.503041 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:47.503049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:47.503408 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.503648 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.504027 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260927 10:47:47.506202 26341 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43695
I20260927 10:47:47.506497 26341 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:47.507198 26341 heartbeater.cc:511] Master 127.10.96.254:43695 requested a full tablet report, sending...
I20260927 10:47:47.508395 26016 ts_manager.cc:194] Registered new tserver with Master: e4c70b545b5f4664a0e16e7a950dd586 (127.10.96.194:39237)
I20260927 10:47:47.508972 26016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:41209
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:47:47.527822 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43695
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:47:47.638679 26346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:47:47.638918 26346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:47:47.639004 26346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:47:47.643138 26346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:47:47.643218 26346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:47:47.643321 26346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:47:47.648551 26346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:47:47.649591 26346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:47:47.650573 26346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:47:47.660912 26354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.660944 26351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:47:47.660921 26352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:47:47.660934 26346 server_base.cc:1034] running on GCE node
I20260927 10:47:47.661657 26346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:47:47.662256 26346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:47:47.663411 26346 hybrid_clock.cc:648] HybridClock initialized: now 1790506067663394 us; error 29 us; skew 500 ppm
Sep 27 10:47:47 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:47:47.666101 26346 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:47:47.666970 26346 webserver.cc:460] Webserver started at http://127.10.96.195:36375/ using document root <none> and password file <none>
I20260927 10:47:47.667508 26346 fs_manager.cc:359] Metadata directory not provided
I20260927 10:47:47.667634 26346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:47:47.667863 26346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:47:47.669862 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "dabaf2d0a5d8464894bbd4d8357be138"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "adff109ab8689b9b4886117ca2e20f69"
server_key_iv: "aa2338e419b936ccca94361a7b3d31b7"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.670415 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "dabaf2d0a5d8464894bbd4d8357be138"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "adff109ab8689b9b4886117ca2e20f69"
server_key_iv: "aa2338e419b936ccca94361a7b3d31b7"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.673938 26346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:47:47.676153 26361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.677325 26346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:47:47.677492 26346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "dabaf2d0a5d8464894bbd4d8357be138"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "adff109ab8689b9b4886117ca2e20f69"
server_key_iv: "aa2338e419b936ccca94361a7b3d31b7"
server_key_version: "encryptionkey@0"
I20260927 10:47:47.677613 26346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:47:47.686506 26346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:47:47.689424 26346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:47:47.689685 26346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:47:47.690246 26346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:47:47.691030 26346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:47:47.691102 26346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.691165 26346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:47:47.691216 26346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:47:47.700438 26346 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40613
I20260927 10:47:47.700492 26474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40613 every 8 connection(s)
I20260927 10:47:47.701367 26346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:47:47.704236 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26346
I20260927 10:47:47.704367 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:47:47.704720 10627 external_mini_cluster.cc:1428] Setting key 87d53ab09242b1b162ac3b5688c82543
Sep 27 10:47:47 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506067, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:47:47.716372 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.703290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:60139 (local address 127.10.96.254:43695)
0927 10:47:47.703461 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.703467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.704448 (+ 981us) server_negotiation.cc:407] Connection header received
0927 10:47:47.706145 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.706149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.706205 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.706283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.707833 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.708385 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.709629 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.709857 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.709971 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.712628 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.712632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.712635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:47.712687 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.714357 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.714870 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.714873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.714874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.714949 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.715260 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.715262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.715263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.715475 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.715553 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.716148 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.716251 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:47:47.717466 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.703686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:60139)
0927 10:47:47.704264 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.704341 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.705910 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.706462 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.706469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.706914 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:47:47.707424 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.707440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.708557 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.708560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.709441 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.709450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.710006 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.710011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.710187 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.710752 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:47:47.710768 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:47:47.712414 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:47:47.714473 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:47.714478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:47.714491 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:47.714767 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:47.715031 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:47:47.715033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:47:47.715035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:47:47.715176 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:47:47.715605 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:47:47.715610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:47:47.715915 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.716102 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.716380 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":114,"threads_started":1}
I20260927 10:47:47.718895 26475 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43695
I20260927 10:47:47.719172 26475 heartbeater.cc:464] Registering TS with master...
I20260927 10:47:47.719687 26475 heartbeater.cc:511] Master 127.10.96.254:43695 requested a full tablet report, sending...
I20260927 10:47:47.720678 26015 ts_manager.cc:194] Registered new tserver with Master: dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613)
I20260927 10:47:47.721289 26015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:60139
I20260927 10:47:47.731895 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:47:47.741199 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.733998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.10.96.254:43695)
0927 10:47:47.734124 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.734128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.734325 (+ 197us) server_negotiation.cc:407] Connection header received
0927 10:47:47.734448 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.734453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.734502 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.734599 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:47:47.735257 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.735767 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.736622 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.736829 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.736904 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.738264 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:47:47.738267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:47:47.738269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:47:47.738291 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:47:47.739552 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.739922 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.739924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.739926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.740034 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:47:47.740324 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:47:47.740328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:47:47.740330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:47:47.740555 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:47:47.740693 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.740927 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.741041 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:47:47.744170 26015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:47:47.747314 26015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:47:47.761580 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.755361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.10.96.195:40613)
0927 10:47:47.756105 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.756119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.756200 (+ 81us) server_negotiation.cc:407] Connection header received
0927 10:47:47.756307 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.756324 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.756568 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.756789 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.757077 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.758311 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.760067 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.760882 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.761013 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.761181 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.761302 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.761400 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":111,"threads_started":1}
I20260927 10:47:47.761596 26490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.755247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.0.0.1:36000)
0927 10:47:47.756475 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.756490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.756570 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.757286 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.757290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.757308 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.757392 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.757398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.758907 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.758914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.760110 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.760118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.761157 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.761160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.761271 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.761294 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.761341 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.761402 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"mutex_wait_us":188,"thread_start_us":96,"threads_started":1}
I20260927 10:47:47.761590 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.755247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40613 (local address 127.0.0.1:56004)
0927 10:47:47.755962 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.756015 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.756154 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.756778 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.756784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.756807 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.756958 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.756965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.758463 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.758466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.759948 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.759959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.761007 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.761010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.761115 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.761158 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.761260 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.761369 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":99,"threads_started":1}
I20260927 10:47:47.762291 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.755924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39237 (local address 127.0.0.1:42466)
0927 10:47:47.756760 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.756772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.756846 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.757499 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.757503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.757519 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.757623 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.757761 (+ 138us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.759006 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.759009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.760788 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.760796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.761900 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.761906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.762016 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.762054 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.762107 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.762176 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":91,"threads_started":1}
I20260927 10:47:47.762388 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.755496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.10.96.193:44649)
0927 10:47:47.756845 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.756860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.756882 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:47:47.756958 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.756963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.757143 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.757292 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.757577 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.758754 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.760219 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.761028 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.761670 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.761837 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.761896 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.762240 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":101,"threads_started":1}
I20260927 10:47:47.763025 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.756322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.10.96.194:39237)
0927 10:47:47.757090 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.757100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.757122 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:47:47.757189 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.757198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.757361 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.757531 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.757874 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.758853 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.760906 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.761728 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.762614 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.762738 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.762784 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.762872 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":109,"threads_started":1}
I20260927 10:47:47.764843 26409 tablet_service.cc:1467] Processing CreateTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d (DEFAULT_TABLE table=test-table [id=9220b432216948c3b63847e8985a2fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:47.765115 26141 tablet_service.cc:1467] Processing CreateTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d (DEFAULT_TABLE table=test-table [id=9220b432216948c3b63847e8985a2fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:47.765926 26409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78a9fff3394620a6e9744ffd4d8b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.765507 26275 tablet_service.cc:1467] Processing CreateTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d (DEFAULT_TABLE table=test-table [id=9220b432216948c3b63847e8985a2fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:47:47.766158 26141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78a9fff3394620a6e9744ffd4d8b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.766496 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78a9fff3394620a6e9744ffd4d8b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:47:47.772017 26492 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Bootstrap starting.
I20260927 10:47:47.773581 26494 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap starting.
I20260927 10:47:47.774575 26492 tablet_bootstrap.cc:654] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.775308 26492 log.cc:824] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.775394 26494 tablet_bootstrap.cc:654] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.775945 26493 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Bootstrap starting.
I20260927 10:47:47.776733 26494 log.cc:824] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.777549 26492 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: No bootstrap required, opened a new log
I20260927 10:47:47.777741 26492 ts_tablet_manager.cc:1397] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:47.778501 26493 tablet_bootstrap.cc:654] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Neither blocks nor log segments found. Creating new log.
I20260927 10:47:47.778535 26494 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: No bootstrap required, opened a new log
I20260927 10:47:47.778723 26494 ts_tablet_manager.cc:1397] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:47:47.779269 26493 log.cc:824] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Log is configured to *not* fsync() on all Append() calls
I20260927 10:47:47.781270 26493 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: No bootstrap required, opened a new log
I20260927 10:47:47.781445 26493 ts_tablet_manager.cc:1397] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:47:47.781749 26492 raft_consensus.cc:348] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.782111 26492 raft_consensus.cc:374] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.782177 26492 raft_consensus.cc:729] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabaf2d0a5d8464894bbd4d8357be138, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.782027 26494 raft_consensus.cc:348] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.782298 26494 raft_consensus.cc:374] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.782357 26494 raft_consensus.cc:729] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf251b4b08fc4ce59de7feae7a00859a, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.782727 26492 consensus_queue.cc:260] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.782807 26494 consensus_queue.cc:260] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.783708 26492 ts_tablet_manager.cc:1428] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:47:47.783828 26494 ts_tablet_manager.cc:1428] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 10:47:47.784237 26207 heartbeater.cc:503] Master 127.10.96.254:43695 was elected leader, sending a full tablet report...
I20260927 10:47:47.784987 26493 raft_consensus.cc:348] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.785173 26475 heartbeater.cc:503] Master 127.10.96.254:43695 was elected leader, sending a full tablet report...
I20260927 10:47:47.785221 26493 raft_consensus.cc:374] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:47:47.785312 26493 raft_consensus.cc:729] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c70b545b5f4664a0e16e7a950dd586, State: Initialized, Role: FOLLOWER
I20260927 10:47:47.785775 26493 consensus_queue.cc:260] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.786660 26493 ts_tablet_manager.cc:1428] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260927 10:47:47.786721 26341 heartbeater.cc:503] Master 127.10.96.254:43695 was elected leader, sending a full tablet report...
W20260927 10:47:47.952464 26476 tablet.cc:2369] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:47.964592 26500 raft_consensus.cc:482] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:47:47.964792 26500 raft_consensus.cc:504] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.965935 26500 leader_election.cc:288] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613), cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:47:47.971163 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.966343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:54261 (local address 127.10.96.195:40613)
0927 10:47:47.966471 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.966476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.966746 (+ 270us) server_negotiation.cc:407] Connection header received
0927 10:47:47.966827 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.966829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.966880 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.966958 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.967368 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.968165 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.969889 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.970487 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.970614 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.970667 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.970959 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.971030 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:47:47.971163 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.966328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:43495 (local address 127.10.96.193:44649)
0927 10:47:47.966558 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.966563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.966591 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:47:47.966783 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.966788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.966850 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.966951 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:47:47.967368 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.968165 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.969887 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.970487 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.970617 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.970667 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.970959 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.971030 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 10:47:47.971202 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.966234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.10.96.194:43495)
0927 10:47:47.966491 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.966507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.966631 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.966983 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.966986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.967016 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.967254 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.967260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.968289 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.968292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.969752 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.969763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.970612 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.970615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.970792 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.970848 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.970928 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.971044 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":176,"spinlock_wait_cycles":15488}
I20260927 10:47:47.971210 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.966234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40613 (local address 127.10.96.194:54261)
0927 10:47:47.966643 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:47.966655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:47:47.966744 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:47:47.966983 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:47:47.966986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:47:47.967011 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:47:47.967252 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.967260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.968289 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.968292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.969756 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:47:47.969763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.970612 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.970615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:47:47.970789 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.970862 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 10:47:47.970928 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 10:47:47.971043 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":402,"spinlock_wait_cycles":1280,"thread_start_us":96,"threads_started":1}
I20260927 10:47:47.971973 26161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78a9fff3394620a6e9744ffd4d8b1d" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf251b4b08fc4ce59de7feae7a00859a" is_pre_election: true
I20260927 10:47:47.971992 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78a9fff3394620a6e9744ffd4d8b1d" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabaf2d0a5d8464894bbd4d8357be138" is_pre_election: true
I20260927 10:47:47.972286 26161 raft_consensus.cc:2454] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 0.
I20260927 10:47:47.972297 26429 raft_consensus.cc:2454] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 0.
I20260927 10:47:47.972810 26229 leader_election.cc:302] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cf251b4b08fc4ce59de7feae7a00859a, dabaf2d0a5d8464894bbd4d8357be138, e4c70b545b5f4664a0e16e7a950dd586; no voters:
I20260927 10:47:47.973097 26500 raft_consensus.cc:2781] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:47:47.973186 26500 raft_consensus.cc:482] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:47:47.973228 26500 raft_consensus.cc:3037] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.974308 26500 raft_consensus.cc:504] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.974676 26500 leader_election.cc:288] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 election: Requested vote from peers dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613), cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:47:47.974979 26429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78a9fff3394620a6e9744ffd4d8b1d" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabaf2d0a5d8464894bbd4d8357be138"
I20260927 10:47:47.975085 26161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78a9fff3394620a6e9744ffd4d8b1d" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf251b4b08fc4ce59de7feae7a00859a"
I20260927 10:47:47.975152 26429 raft_consensus.cc:3037] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.975230 26161 raft_consensus.cc:3037] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:47:47.976239 26429 raft_consensus.cc:2454] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 1.
I20260927 10:47:47.976248 26161 raft_consensus.cc:2454] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 1.
I20260927 10:47:47.976599 26231 leader_election.cc:302] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf251b4b08fc4ce59de7feae7a00859a, e4c70b545b5f4664a0e16e7a950dd586; no voters:
I20260927 10:47:47.976809 26500 raft_consensus.cc:2781] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:47:47.977046 26500 raft_consensus.cc:686] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 1 LEADER]: Becoming Leader. State: Replica: e4c70b545b5f4664a0e16e7a950dd586, State: Running, Role: LEADER
I20260927 10:47:47.977392 26500 consensus_queue.cc:237] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:47:47.980964 26015 catalog_manager.cc:5495] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c70b545b5f4664a0e16e7a950dd586 (127.10.96.194). New cstate: current_term: 1 leader_uuid: "e4c70b545b5f4664a0e16e7a950dd586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } health_report { overall_health: HEALTHY } } }
W20260927 10:47:47.990465 26342 tablet.cc:2369] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:47:47.994998 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:47.991270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.10.96.194:39237)
0927 10:47:47.991379 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 10:47:47.991383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:47:47.991396 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:47:47.991536 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:47:47.991539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:47:47.991610 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:47:47.991695 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:47:47.991929 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.992418 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.993604 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:47:47.993811 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:47:47.993888 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:47:47.994206 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:47:47.994609 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:47:47.994696 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 10:47:47.994788 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 10:47:47.994838 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:47:48.000730 26161 raft_consensus.cc:1261] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Refusing update from remote peer e4c70b545b5f4664a0e16e7a950dd586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:48.000756 26429 raft_consensus.cc:1261] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Refusing update from remote peer e4c70b545b5f4664a0e16e7a950dd586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:47:48.001504 26502 consensus_queue.cc:1035] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:48.001718 26500 consensus_queue.cc:1035] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:47:48.008566 26505 mvcc.cc:205] Tried to move back new op lower bound from 7333912854523580416 to 7333912854438621184. Current Snapshot: MvccSnapshot[applied={T|T < 7333912854523580416}]
I20260927 10:47:48.010854 26506 mvcc.cc:205] Tried to move back new op lower bound from 7333912854523580416 to 7333912854438621184. Current Snapshot: MvccSnapshot[applied={T|T < 7333912854523580416}]
I20260927 10:47:48.018093 26015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:48.018285 26015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:47:48.020851 26015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Sending DeleteTablet for 3 replicas of tablet be78a9fff3394620a6e9744ffd4d8b1d
I20260927 10:47:48.021689 26141 tablet_service.cc:1514] Processing DeleteTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:48 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:36000
I20260927 10:47:48.021827 26275 tablet_service.cc:1514] Processing DeleteTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:48 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:42466
I20260927 10:47:48.021852 26409 tablet_service.cc:1514] Processing DeleteTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:48 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:56004
I20260927 10:47:48.022377 26515 tablet_replica.cc:330] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: stopping tablet replica
I20260927 10:47:48.022514 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26078
I20260927 10:47:48.022706 26515 raft_consensus.cc:2229] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:48.022984 26516 tablet_replica.cc:330] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: stopping tablet replica
I20260927 10:47:48.023301 26516 raft_consensus.cc:2229] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:47:48.023614 26516 raft_consensus.cc:2258] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:48.023991 26517 tablet_replica.cc:330] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: stopping tablet replica
I20260927 10:47:48.024279 26517 raft_consensus.cc:2229] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:47:48.024627 26517 raft_consensus.cc:2258] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:47:48.026273 26516 ts_tablet_manager.cc:1905] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:48.026753 26517 ts_tablet_manager.cc:1905] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:47:48.029177 26516 ts_tablet_manager.cc:1918] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:48.029356 26516 log.cc:1196] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/be78a9fff3394620a6e9744ffd4d8b1d
I20260927 10:47:48.029775 26516 ts_tablet_manager.cc:1939] T be78a9fff3394620a6e9744ffd4d8b1d P dabaf2d0a5d8464894bbd4d8357be138: Deleting consensus metadata
I20260927 10:47:48.030262 26517 ts_tablet_manager.cc:1918] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:47:48.030386 26517 log.cc:1196] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/be78a9fff3394620a6e9744ffd4d8b1d
W20260927 10:47:48.030653 26003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:44649 (error 108)
I20260927 10:47:48.030709 26517 ts_tablet_manager.cc:1939] T be78a9fff3394620a6e9744ffd4d8b1d P e4c70b545b5f4664a0e16e7a950dd586: Deleting consensus metadata
I20260927 10:47:48.030913 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26212
I20260927 10:47:48.032873 26001 catalog_manager.cc:4841] TS dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613): tablet be78a9fff3394620a6e9744ffd4d8b1d (table test-table [id=9220b432216948c3b63847e8985a2fc7]) successfully deleted
I20260927 10:47:48.033491 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:48.033135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.0.0.1:36006)
0927 10:47:48.033269 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:48.033350 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 10:47:48.033634 26003 catalog_manager.cc:4584] TS cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be78a9fff3394620a6e9744ffd4d8b1d: Network error: Client connection negotiation failed: client connection to 127.10.96.193:44649: connect: Connection refused (error 111)
I20260927 10:47:48.038268 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26346
I20260927 10:47:48.039629 26486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:47:48.038395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39237 (local address 127.0.0.1:42472)
0927 10:47:48.039435 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0927 10:47:48.039513 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":943}
I20260927 10:47:48.043979 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 25984
I20260927 10:48:08.050689 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43695
--webserver_interface=127.10.96.254
--webserver_port=36889
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:08.163616 26519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:08.163901 26519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:08.163944 26519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:08.168102 26519 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:48:08.168164 26519 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:48:08.168179 26519 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:48:08.168195 26519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:08.168213 26519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:08.168273 26519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:08.168306 26519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:08.168327 26519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:08.173738 26519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:08.174865 26519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:08.175935 26519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:48:08.189191 26519 server_base.cc:1034] running on GCE node
W20260927 10:48:08.189368 26525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.189507 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.189527 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:08.190275 26519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:08.191501 26519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:08.192705 26519 hybrid_clock.cc:648] HybridClock initialized: now 1790506088192684 us; error 39 us; skew 500 ppm
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:08.199082 26519 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:48:08.200048 26519 webserver.cc:460] Webserver started at http://127.10.96.254:36889/ using document root <none> and password file <none>
I20260927 10:48:08.200747 26519 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:08.200867 26519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:08.204893 26519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:08.207188 26535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.208489 26519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:48:08.208732 26519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "6d6fe792107e433f9f0abcda363c876b"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-wmg0"
server_key: "3261ea40033a140f4500e04b8bdcb31c"
server_key_iv: "cbd6370e5bcd0f429ee3c2b8f02d0bda"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.209343 26519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:08.228595 26519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:08.231714 26519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:08.232033 26519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:08.239528 26519 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43695
I20260927 10:48:08.239823 26587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43695 every 8 connection(s)
I20260927 10:48:08.240676 26519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:08.244978 26588 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:48:08.247112 26588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap starting.
I20260927 10:48:08.247479 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26519
I20260927 10:48:08.247889 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:44649
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:43695
--webserver_port=40431
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:08.270762 26588 log.cc:824] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:08.281847 26588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:48:08.282284 26588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap complete.
I20260927 10:48:08.287775 26588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:08.288298 26588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6fe792107e433f9f0abcda363c876b, State: Initialized, Role: FOLLOWER
I20260927 10:48:08.288853 26588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:08.288995 26588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:08.289068 26588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:08.289165 26588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:48:08.291829 26588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:08.292234 26588 leader_election.cc:302] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6fe792107e433f9f0abcda363c876b; no voters:
I20260927 10:48:08.292579 26588 leader_election.cc:288] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 10:48:08.292706 26592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:48:08.293006 26592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 LEADER]: Becoming Leader. State: Replica: 6d6fe792107e433f9f0abcda363c876b, State: Running, Role: LEADER
I20260927 10:48:08.293378 26592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:08.293896 26588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:08.294824 26593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6d6fe792107e433f9f0abcda363c876b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } } }
I20260927 10:48:08.294937 26593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:08.295347 26594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d6fe792107e433f9f0abcda363c876b. Latest consensus state: current_term: 2 leader_uuid: "6d6fe792107e433f9f0abcda363c876b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } } }
I20260927 10:48:08.295440 26594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:08.296034 26601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:08.301059 26601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:08.301728 26601 catalog_manager.cc:1259] Loaded cluster ID: 1428d134ddfe4db885055ebed40fa13b
I20260927 10:48:08.301805 26601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:08.304386 26601 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:08.305094 26601 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Loaded TSK: 0
I20260927 10:48:08.325695 26601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Generated new TSK 1
I20260927 10:48:08.326125 26601 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:48:08.379303 26590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:08.379604 26590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:08.379679 26590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:08.383769 26590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:08.383857 26590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:08.383946 26590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:08.389343 26590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:44649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=40431
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:08.390444 26590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:08.391464 26590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:08.401530 26618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.401484 26616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.401487 26615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:08.401885 26590 server_base.cc:1034] running on GCE node
I20260927 10:48:08.402339 26590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:08.403115 26590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:08.404311 26590 hybrid_clock.cc:648] HybridClock initialized: now 1790506088404303 us; error 48 us; skew 500 ppm
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:08.407891 26590 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:48:08.408805 26590 webserver.cc:460] Webserver started at http://127.10.96.193:40431/ using document root <none> and password file <none>
I20260927 10:48:08.409389 26590 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:08.409516 26590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:08.413709 26590 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 10:48:08.416119 26625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.417652 26590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:08.417796 26590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "cf251b4b08fc4ce59de7feae7a00859a"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "d5c9e4b3e48325ea8aea40733f9d2522"
server_key_iv: "7eab34e6dd5b3dfd49752ad1a03bb204"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.418370 26590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:08.435109 26590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:08.438736 26590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:08.438939 26590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:08.439584 26590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:08.440665 26632 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:48:08.446031 26590 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:48:08.446130 26590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 10:48:08.446220 26590 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:48:08.447997 26590 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:48:08.448072 26590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:48:08.448369 26632 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap starting.
I20260927 10:48:08.458732 26590 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:44649
I20260927 10:48:08.458803 26739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:44649 every 8 connection(s)
I20260927 10:48:08.459710 26590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:08.465613 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26590
I20260927 10:48:08.466080 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:39237
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:43695
--webserver_port=39613
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:08.467459 26632 log.cc:824] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:08.475775 26632 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:48:08.476238 26632 tablet_bootstrap.cc:492] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap complete.
I20260927 10:48:08.477605 26632 ts_tablet_manager.cc:1397] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.003s
I20260927 10:48:08.479629 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.461788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:38061)
0927 10:48:08.462484 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:08.462541 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:08.463723 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:08.464962 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:08.464977 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:08.465589 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:08.466233 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.466254 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.468113 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.468115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.469313 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.469324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.470490 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.470494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.470680 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.471424 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:08.471445 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:08.473485 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:08.476070 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.476076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.476100 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.476525 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.476969 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.476974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.476978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.477178 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.477911 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:08.477918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:08.478199 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 10:48:08.478388 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 10:48:08.478715 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":158,"threads_started":1}
I20260927 10:48:08.479648 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.461788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:38061 (local address 127.10.96.254:43695)
0927 10:48:08.462212 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.462232 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.462672 (+ 440us) server_negotiation.cc:407] Connection header received
0927 10:48:08.464187 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.464198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.464517 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.464790 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.466445 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.467939 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.469450 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.470320 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.470525 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.473707 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.473713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.473728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:08.473762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.475939 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.476667 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.476676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.476685 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.476856 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.477289 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.477295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.477300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.477789 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.477927 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.478554 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.478858 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
I20260927 10:48:08.480464 26632 raft_consensus.cc:348] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:48:08.480898 26632 raft_consensus.cc:729] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf251b4b08fc4ce59de7feae7a00859a, State: Initialized, Role: FOLLOWER
I20260927 10:48:08.481401 26632 consensus_queue.cc:260] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } } peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } }
I20260927 10:48:08.482415 26740 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43695
I20260927 10:48:08.482460 26632 ts_tablet_manager.cc:1428] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:48:08.482625 26740 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:08.483074 26740 heartbeater.cc:511] Master 127.10.96.254:43695 requested a full tablet report, sending...
I20260927 10:48:08.485570 26552 ts_manager.cc:194] Registered new tserver with Master: cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:48:08.488874 26552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:38061
I20260927 10:48:08.491273 26740 heartbeater.cc:503] Master 127.10.96.254:43695 was elected leader, sending a full tablet report...
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:48:08.499634 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.488204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.0.0.1:41342)
0927 10:48:08.488971 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:08.489017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:08.489195 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:08.489886 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:08.489891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:08.490313 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:08.490437 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.490444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.491690 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.491693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.492627 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.492638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.493371 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.493373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.493448 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.494116 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:08.494127 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:08.495779 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:08.497857 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.497868 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.497874 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.498109 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.498414 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.498417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.498419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.498505 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.498970 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:08.498976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:08.499102 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 10:48:08.499248 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 10:48:08.499449 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":123,"threads_started":1}
I20260927 10:48:08.499678 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.488299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.10.96.193:44649)
0927 10:48:08.488706 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.488719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.489096 (+ 377us) server_negotiation.cc:407] Connection header received
0927 10:48:08.489366 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.489371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.489590 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.489693 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.490566 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.491556 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.492764 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.493171 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.493234 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.495929 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.495937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.495947 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:08.495967 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.497746 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.498233 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.498240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.498241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.498315 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.498595 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.498600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.498602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.498838 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.498982 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.499338 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.499533 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":119,"threads_started":1}
I20260927 10:48:08.500773 26674 tablet_service.cc:1514] Processing DeleteTablet for tablet be78a9fff3394620a6e9744ffd4d8b1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:47:48 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:41342
I20260927 10:48:08.501762 26749 tablet_replica.cc:330] stopping tablet replica
I20260927 10:48:08.502086 26749 raft_consensus.cc:2229] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:08.502364 26749 raft_consensus.cc:2258] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:08.503464 26749 ts_tablet_manager.cc:1905] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:08.508268 26749 ts_tablet_manager.cc:1918] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:08.508369 26749 log.cc:1196] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal/wals/be78a9fff3394620a6e9744ffd4d8b1d
I20260927 10:48:08.508630 26749 ts_tablet_manager.cc:1939] T be78a9fff3394620a6e9744ffd4d8b1d P cf251b4b08fc4ce59de7feae7a00859a: Deleting consensus metadata
I20260927 10:48:08.510695 26539 catalog_manager.cc:4841] TS cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649): tablet be78a9fff3394620a6e9744ffd4d8b1d (table test-table [id=9220b432216948c3b63847e8985a2fc7]) successfully deleted
W20260927 10:48:08.585899 26744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:08.586149 26744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:08.586222 26744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:08.590195 26744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:08.590247 26744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:08.590348 26744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:08.595561 26744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=39613
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:08.596633 26744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:08.597853 26744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:08.607349 26755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.607353 26757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.607353 26754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:08.607656 26744 server_base.cc:1034] running on GCE node
I20260927 10:48:08.608080 26744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:08.608671 26744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:08.609830 26744 hybrid_clock.cc:648] HybridClock initialized: now 1790506088609818 us; error 25 us; skew 500 ppm
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:08.612552 26744 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:48:08.613442 26744 webserver.cc:460] Webserver started at http://127.10.96.194:39613/ using document root <none> and password file <none>
I20260927 10:48:08.614032 26744 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:08.614133 26744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:08.617830 26744 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:48:08.619844 26764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.621050 26744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:08.621174 26744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "e4c70b545b5f4664a0e16e7a950dd586"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "5ba267274247c06317d2b9ab7ad8bdba"
server_key_iv: "f5a6eb2d63e07d5b125175550539c66f"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.621754 26744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:08.645537 26744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:08.649111 26744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:08.649324 26744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:08.649976 26744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:08.650846 26744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:08.650914 26744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.650978 26744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:08.651018 26744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.660463 26744 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:39237
I20260927 10:48:08.660485 26877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:39237 every 8 connection(s)
I20260927 10:48:08.661598 26744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:08.662912 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26744
I20260927 10:48:08.663388 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:40613
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:43695
--webserver_port=36375
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:08.676002 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.663629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:50459 (local address 127.10.96.254:43695)
0927 10:48:08.663824 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.663831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.664883 (+ 1052us) server_negotiation.cc:407] Connection header received
0927 10:48:08.666408 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.666412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.666475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.666582 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.667846 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.668381 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.669644 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.669882 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.669994 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.672414 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.672417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.672420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:08.672444 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.673946 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.674459 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.674464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.674466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.674548 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.674898 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.674901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.674903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.675086 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.675190 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.675682 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.675825 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 10:48:08.676831 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.664114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:50459)
0927 10:48:08.664680 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:08.664751 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:08.666189 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:08.666765 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:08.666771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:08.667137 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:08.667693 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.667707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.668514 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.668523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.669523 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.669533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.670033 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.670037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.670208 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.670815 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:08.670832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:08.672242 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:08.674075 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.674080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.674095 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.674355 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.674638 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.674642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.674644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.674794 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.675187 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:08.675194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:08.675443 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 10:48:08.675627 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 10:48:08.675964 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260927 10:48:08.678251 26878 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43695
I20260927 10:48:08.678512 26878 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:08.679019 26878 heartbeater.cc:511] Master 127.10.96.254:43695 requested a full tablet report, sending...
I20260927 10:48:08.680124 26552 ts_manager.cc:194] Registered new tserver with Master: e4c70b545b5f4664a0e16e7a950dd586 (127.10.96.194:39237)
I20260927 10:48:08.680892 26552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:50459
W20260927 10:48:08.778587 26881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:08.778851 26881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:08.778894 26881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:08.782950 26881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:08.783010 26881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:08.783068 26881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:08.788259 26881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:40613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=36375
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:08.789252 26881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:08.790372 26881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:08.800113 26889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.800099 26886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:08.800124 26887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:08.800618 26881 server_base.cc:1034] running on GCE node
I20260927 10:48:08.801090 26881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:08.801884 26881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:08.803066 26881 hybrid_clock.cc:648] HybridClock initialized: now 1790506088803053 us; error 31 us; skew 500 ppm
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:08.805926 26881 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:48:08.806818 26881 webserver.cc:460] Webserver started at http://127.10.96.195:36375/ using document root <none> and password file <none>
I20260927 10:48:08.807391 26881 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:08.807493 26881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:08.811419 26881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:08.813534 26896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.814762 26881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:08.814903 26881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "dabaf2d0a5d8464894bbd4d8357be138"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "adff109ab8689b9b4886117ca2e20f69"
server_key_iv: "aa2338e419b936ccca94361a7b3d31b7"
server_key_version: "encryptionkey@0"
I20260927 10:48:08.815434 26881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:08.842096 26881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:08.845651 26881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:08.845861 26881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:08.846483 26881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:08.847319 26881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:08.847393 26881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.847464 26881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:08.847514 26881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:08.856994 26881 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40613
I20260927 10:48:08.857014 27009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40613 every 8 connection(s)
I20260927 10:48:08.858095 26881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:08.860643 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 26881
Sep 27 10:48:08 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506088, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:08.870208 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.861303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.10.96.254:43695)
0927 10:48:08.861572 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.861576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.861763 (+ 187us) server_negotiation.cc:407] Connection header received
0927 10:48:08.861940 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.861943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.861997 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.862090 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.862922 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.863455 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.864520 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.864798 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.864934 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.866549 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.866553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.866555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:08.866584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.868245 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.868740 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.868747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.868749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.868845 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.869165 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.869168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.869170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.869361 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.869608 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.869807 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.869986 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260927 10:48:08.873011 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.859915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:50767 (local address 127.10.96.254:43695)
0927 10:48:08.860066 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.860070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.861071 (+ 1001us) server_negotiation.cc:407] Connection header received
0927 10:48:08.862595 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.862599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.862645 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.862732 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.863968 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.864653 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.866025 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.866400 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.866513 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.869271 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.869274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.869277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:08.869315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.870979 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.871571 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.871574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.871576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.871658 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.872032 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.872035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.872036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.872228 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.872311 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.872822 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.872908 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:48:08.873977 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.860297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:50767)
0927 10:48:08.860892 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:08.860938 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:08.862384 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:08.862969 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:08.862977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:08.863326 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:08.863830 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.863843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.864773 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.864777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.865858 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.865868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.866566 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.866570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.866789 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.867281 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:08.867303 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:08.869106 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:08.871113 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.871118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.871150 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.871476 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.871746 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:08.871748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:08.871750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:08.871933 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:08.872350 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:08.872356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:08.872594 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 10:48:08.872775 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 10:48:08.873035 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":152,"threads_started":1}
I20260927 10:48:08.875102 27010 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43695
I20260927 10:48:08.875336 27010 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:08.875777 27010 heartbeater.cc:511] Master 127.10.96.254:43695 requested a full tablet report, sending...
I20260927 10:48:08.876732 26552 ts_manager.cc:194] Registered new tserver with Master: dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613)
I20260927 10:48:08.877295 26552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:50767
I20260927 10:48:08.883961 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:08.888007 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.884654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.10.96.254:43695)
0927 10:48:08.884798 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.884803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.884973 (+ 170us) server_negotiation.cc:407] Connection header received
0927 10:48:08.885109 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.885112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.885209 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.885301 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:08.885595 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.886111 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.886988 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.887200 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.887288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.887573 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:08.887705 (+ 132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 10:48:08.887910 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55518: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 10:48:08.888110 26742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55518: token expired
W20260927 10:48:08.888501 26478 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.96.254:43695)
I20260927 10:48:08.895874 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.888882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.10.96.254:43695)
0927 10:48:08.889099 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.889102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.889113 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:48:08.889141 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.889143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.889177 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.889262 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:08.889956 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.890439 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.891242 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.891473 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.891559 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.892877 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:08.892880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:08.892882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:08.892903 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:08.894460 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.894849 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.894851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.894853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.894924 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:08.895147 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:08.895149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:08.895150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:08.895334 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:08.895473 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.895674 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.895762 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260927 10:48:08.898447 26552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:08.901752 26552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:08.909392 26674 tablet_service.cc:1467] Processing CreateTablet for tablet 88ed9b114d3a439fab6d9cdd106b8a53 (DEFAULT_TABLE table=test-table [id=620a51dea1f443f89d9333df4952cc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:08.910094 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed9b114d3a439fab6d9cdd106b8a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:08.912916 26632 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap starting.
I20260927 10:48:08.913826 26632 tablet_bootstrap.cc:654] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:08.914422 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.908362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39237 (local address 127.0.0.1:37796)
0927 10:48:08.908504 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:08.908524 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:08.908586 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:08.910076 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:08.910079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:08.910096 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:08.910208 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.910214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.911614 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.911620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.912745 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.912755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.913953 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.913957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.914063 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.914122 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 10:48:08.914187 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 10:48:08.914280 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:48:08.914644 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.909093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.10.96.195:40613)
0927 10:48:08.909415 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.909430 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.909688 (+ 258us) server_negotiation.cc:407] Connection header received
0927 10:48:08.909897 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.909903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.910075 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.910206 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:08.910536 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.911579 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.913000 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.913811 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.913911 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.914071 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.914393 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.914506 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260927 10:48:08.914943 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.908865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.10.96.194:39237)
0927 10:48:08.909503 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:08.909516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:08.909545 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:48:08.909635 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:08.909643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:08.909914 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:08.910083 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:08.910365 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.911488 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.912906 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.913810 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.914522 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.914660 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:08.914715 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 10:48:08.914814 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":124,"threads_started":1}
I20260927 10:48:08.915198 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:08.908806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40613 (local address 127.0.0.1:43594)
0927 10:48:08.909596 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:08.909609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:08.909755 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:08.910287 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:08.910290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:08.910307 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:08.910393 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.910398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.911674 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.911677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.912858 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:08.912868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:08.914168 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:08.914177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:08.914255 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:08.914271 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 10:48:08.914357 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 10:48:08.914426 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":112,"threads_started":1}
I20260927 10:48:08.915354 26632 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: No bootstrap required, opened a new log
I20260927 10:48:08.915444 26632 ts_tablet_manager.cc:1397] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:08.915861 26632 raft_consensus.cc:348] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:08.915980 26632 raft_consensus.cc:374] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:08.916028 26632 raft_consensus.cc:729] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf251b4b08fc4ce59de7feae7a00859a, State: Initialized, Role: FOLLOWER
I20260927 10:48:08.916160 26632 consensus_queue.cc:260] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:08.916481 26632 ts_tablet_manager.cc:1428] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 10:48:08.918325 26812 tablet_service.cc:1467] Processing CreateTablet for tablet 88ed9b114d3a439fab6d9cdd106b8a53 (DEFAULT_TABLE table=test-table [id=620a51dea1f443f89d9333df4952cc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:08.918694 26943 tablet_service.cc:1467] Processing CreateTablet for tablet 88ed9b114d3a439fab6d9cdd106b8a53 (DEFAULT_TABLE table=test-table [id=620a51dea1f443f89d9333df4952cc93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:08.919554 26812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed9b114d3a439fab6d9cdd106b8a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:08.919904 26943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed9b114d3a439fab6d9cdd106b8a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:08.926651 27023 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Bootstrap starting.
I20260927 10:48:08.926700 27024 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Bootstrap starting.
I20260927 10:48:08.929032 27024 tablet_bootstrap.cc:654] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:08.929045 27023 tablet_bootstrap.cc:654] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:08.929819 27023 log.cc:824] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:08.929819 27024 log.cc:824] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:08.931677 27024 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: No bootstrap required, opened a new log
I20260927 10:48:08.931875 27024 ts_tablet_manager.cc:1397] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:48:08.932030 27023 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: No bootstrap required, opened a new log
I20260927 10:48:08.932286 27023 ts_tablet_manager.cc:1397] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:08.935420 27024 raft_consensus.cc:348] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:08.935642 27024 raft_consensus.cc:374] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:08.935706 27024 raft_consensus.cc:729] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabaf2d0a5d8464894bbd4d8357be138, State: Initialized, Role: FOLLOWER
I20260927 10:48:08.935529 27023 raft_consensus.cc:348] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:08.935788 27023 raft_consensus.cc:374] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:08.935846 27023 raft_consensus.cc:729] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c70b545b5f4664a0e16e7a950dd586, State: Initialized, Role: FOLLOWER
I20260927 10:48:08.936157 27024 consensus_queue.cc:260] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:08.936242 27023 consensus_queue.cc:260] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:08.936846 27010 heartbeater.cc:503] Master 127.10.96.254:43695 was elected leader, sending a full tablet report...
I20260927 10:48:08.936904 26878 heartbeater.cc:503] Master 127.10.96.254:43695 was elected leader, sending a full tablet report...
I20260927 10:48:08.937211 27024 ts_tablet_manager.cc:1428] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260927 10:48:08.937211 27023 ts_tablet_manager.cc:1428] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
W20260927 10:48:08.960965 26741 tablet.cc:2369] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:09.055904 27028 raft_consensus.cc:482] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:09.056190 27028 raft_consensus.cc:504] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:09.057440 27028 leader_election.cc:288] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613), cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:48:09.062079 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.057874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:40211 (local address 127.10.96.195:40613)
0927 10:48:09.058024 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:09.058031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:09.058321 (+ 290us) server_negotiation.cc:407] Connection header received
0927 10:48:09.058440 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:09.058443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:09.058517 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:09.058597 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:09.058892 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.059506 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.061112 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.061526 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.061656 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:09.061707 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:09.061888 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:48:09.061954 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:48:09.062103 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.057792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40613 (local address 127.10.96.194:40211)
0927 10:48:09.058210 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:09.058225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:09.058334 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:09.058638 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:09.058641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:09.058655 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:09.058782 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:09.058788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.059592 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.059595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:09.060988 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:09.060999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.061648 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.061652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:09.061751 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:09.061780 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 10:48:09.061874 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 10:48:09.061948 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":110,"thread_start_us":128,"threads_started":1}
I20260927 10:48:09.062517 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.057792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.10.96.194:45917)
0927 10:48:09.058030 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:09.058046 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:09.058153 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:09.058529 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:09.058533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:09.058553 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:09.058689 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:09.058705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.059548 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.059551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:09.060988 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:09.060999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.062118 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.062122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:09.062227 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:09.062247 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:48:09.062309 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 10:48:09.062402 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":94}
I20260927 10:48:09.062559 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.057874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:45917 (local address 127.10.96.193:44649)
0927 10:48:09.058178 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:09.058183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:09.058249 (+ 66us) server_negotiation.cc:407] Connection header received
0927 10:48:09.058313 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:09.058316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:09.058382 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:09.058490 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:09.058822 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.059421 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.061551 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.061979 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.062130 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:09.062317 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:09.062357 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 10:48:09.062427 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":97,"threads_started":1}
I20260927 10:48:09.062906 26964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed9b114d3a439fab6d9cdd106b8a53" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabaf2d0a5d8464894bbd4d8357be138" is_pre_election: true
I20260927 10:48:09.063268 26964 raft_consensus.cc:2454] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 0.
I20260927 10:48:09.063294 26694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed9b114d3a439fab6d9cdd106b8a53" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf251b4b08fc4ce59de7feae7a00859a" is_pre_election: true
I20260927 10:48:09.063604 26694 raft_consensus.cc:2454] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 0.
I20260927 10:48:09.063979 26766 leader_election.cc:302] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabaf2d0a5d8464894bbd4d8357be138, e4c70b545b5f4664a0e16e7a950dd586; no voters:
I20260927 10:48:09.064333 27028 raft_consensus.cc:2781] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:09.064414 27028 raft_consensus.cc:482] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:09.064445 27028 raft_consensus.cc:3037] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:09.065523 27028 raft_consensus.cc:504] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:09.065909 27028 leader_election.cc:288] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 election: Requested vote from peers dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613), cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:48:09.066236 26964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed9b114d3a439fab6d9cdd106b8a53" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabaf2d0a5d8464894bbd4d8357be138"
I20260927 10:48:09.066331 26694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed9b114d3a439fab6d9cdd106b8a53" candidate_uuid: "e4c70b545b5f4664a0e16e7a950dd586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf251b4b08fc4ce59de7feae7a00859a"
I20260927 10:48:09.066386 26964 raft_consensus.cc:3037] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:09.066509 26694 raft_consensus.cc:3037] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:09.067458 26964 raft_consensus.cc:2454] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 1.
I20260927 10:48:09.067544 26694 raft_consensus.cc:2454] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c70b545b5f4664a0e16e7a950dd586 in term 1.
I20260927 10:48:09.067766 26766 leader_election.cc:302] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabaf2d0a5d8464894bbd4d8357be138, e4c70b545b5f4664a0e16e7a950dd586; no voters:
I20260927 10:48:09.067929 27028 raft_consensus.cc:2781] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:09.068148 27028 raft_consensus.cc:686] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 LEADER]: Becoming Leader. State: Replica: e4c70b545b5f4664a0e16e7a950dd586, State: Running, Role: LEADER
I20260927 10:48:09.068542 27028 consensus_queue.cc:237] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:09.071820 26552 catalog_manager.cc:5495] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c70b545b5f4664a0e16e7a950dd586 (127.10.96.194). New cstate: current_term: 1 leader_uuid: "e4c70b545b5f4664a0e16e7a950dd586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } health_report { overall_health: UNKNOWN } } }
I20260927 10:48:09.104784 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.101174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.10.96.194:39237)
0927 10:48:09.101316 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:09.101320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:09.101333 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:48:09.101510 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:09.101513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:09.101571 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:09.101662 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:09.101869 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.102455 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.103248 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:09.103492 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:09.103581 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:09.103894 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:09.104331 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:09.104437 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:09.104513 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 10:48:09.104562 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 10:48:09.109009 27011 tablet.cc:2369] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:09.110601 26694 raft_consensus.cc:1261] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Refusing update from remote peer e4c70b545b5f4664a0e16e7a950dd586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:09.110605 26964 raft_consensus.cc:1261] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Refusing update from remote peer e4c70b545b5f4664a0e16e7a950dd586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:09.111297 27031 consensus_queue.cc:1035] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:09.111497 27028 consensus_queue.cc:1035] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:09.125989 26551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:09.126179 26551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:09.128741 26551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Sending DeleteTablet for 3 replicas of tablet 88ed9b114d3a439fab6d9cdd106b8a53
I20260927 10:48:09.129407 26812 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ed9b114d3a439fab6d9cdd106b8a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:09 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:37796
I20260927 10:48:09.129527 26943 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ed9b114d3a439fab6d9cdd106b8a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:09 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:43594
I20260927 10:48:09.129865 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26590
I20260927 10:48:09.129881 26674 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ed9b114d3a439fab6d9cdd106b8a53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:09 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:41342
I20260927 10:48:09.131016 27043 tablet_replica.cc:330] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: stopping tablet replica
I20260927 10:48:09.131397 27043 raft_consensus.cc:2229] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:09.131721 27043 raft_consensus.cc:2258] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:09.133502 27043 ts_tablet_manager.cc:1905] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:09.136984 27043 ts_tablet_manager.cc:1918] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:09.137094 27043 log.cc:1196] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/88ed9b114d3a439fab6d9cdd106b8a53
I20260927 10:48:09.137424 27042 tablet_replica.cc:330] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: stopping tablet replica
I20260927 10:48:09.137552 27043 ts_tablet_manager.cc:1939] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Deleting consensus metadata
I20260927 10:48:09.137833 27042 raft_consensus.cc:2229] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 LEADER]: Raft consensus shutting down.
W20260927 10:48:09.138054 26539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:44649 (error 108)
I20260927 10:48:09.138252 27042 raft_consensus.cc:2258] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:09.139034 26537 catalog_manager.cc:4841] TS dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613): tablet 88ed9b114d3a439fab6d9cdd106b8a53 (table test-table [id=620a51dea1f443f89d9333df4952cc93]) successfully deleted
I20260927 10:48:09.139806 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26744
I20260927 10:48:09.139863 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.139560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.0.0.1:41358)
0927 10:48:09.139687 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:09.139764 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:48:09.140022 26539 catalog_manager.cc:4584] TS cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88ed9b114d3a439fab6d9cdd106b8a53: Network error: Client connection negotiation failed: client connection to 127.10.96.193:44649: connect: Connection refused (error 111)
I20260927 10:48:09.146191 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26881
I20260927 10:48:09.146528 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:09.146160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39237 (local address 127.0.0.1:37802)
0927 10:48:09.146366 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:09.146432 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:39237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 10:48:09.151811 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 26519
I20260927 10:48:29.158329 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43695
--webserver_interface=127.10.96.254
--webserver_port=36889
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:29.273350 27045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:29.273811 27045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:29.273891 27045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:29.278210 27045 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 10:48:29.278292 27045 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 10:48:29.278321 27045 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 10:48:29.278357 27045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:29.278395 27045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:29.278435 27045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:29.278471 27045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:29.278506 27045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:29.284164 27045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:29.285324 27045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:29.286455 27045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:48:29.295262 27045 server_base.cc:1034] running on GCE node
W20260927 10:48:29.295197 27053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.295214 27050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.295197 27051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:29.296201 27045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:29.297343 27045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:29.298513 27045 hybrid_clock.cc:648] HybridClock initialized: now 1790506109298496 us; error 33 us; skew 500 ppm
Sep 27 10:48:29 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:29.307097 27045 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:48:29.308171 27045 webserver.cc:460] Webserver started at http://127.10.96.254:36889/ using document root <none> and password file <none>
I20260927 10:48:29.308785 27045 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:29.308851 27045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:29.312906 27045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:29.315269 27060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:29.316567 27045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:48:29.316717 27045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "6d6fe792107e433f9f0abcda363c876b"
format_stamp: "Formatted at 2026-09-27 10:47:46 on dist-test-slave-wmg0"
server_key: "3261ea40033a140f4500e04b8bdcb31c"
server_key_iv: "cbd6370e5bcd0f429ee3c2b8f02d0bda"
server_key_version: "encryptionkey@0"
I20260927 10:48:29.317265 27045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:29.330204 27045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:29.333227 27045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:29.333444 27045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:29.340793 27045 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43695
I20260927 10:48:29.341106 27112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43695 every 8 connection(s)
I20260927 10:48:29.341996 27045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:29.345332 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27045
I20260927 10:48:29.345846 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:44649
--local_ip_for_outbound_sockets=127.10.96.193
--tserver_master_addrs=127.10.96.254:43695
--webserver_port=40431
--webserver_interface=127.10.96.193
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:29.346349 27113 sys_catalog.cc:263] Verifying existing consensus state
I20260927 10:48:29.348501 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap starting.
I20260927 10:48:29.373441 27113 log.cc:824] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:29.387722 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:48:29.388211 27113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: Bootstrap complete.
I20260927 10:48:29.393812 27113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:29.394341 27113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d6fe792107e433f9f0abcda363c876b, State: Initialized, Role: FOLLOWER
I20260927 10:48:29.394902 27113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:29.395047 27113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:29.395117 27113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:29.395207 27113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 2 FOLLOWER]: Advancing to term 3
I20260927 10:48:29.397816 27113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:29.398185 27113 leader_election.cc:302] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d6fe792107e433f9f0abcda363c876b; no voters:
I20260927 10:48:29.398540 27113 leader_election.cc:288] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 10:48:29.398675 27117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 10:48:29.398974 27117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [term 3 LEADER]: Becoming Leader. State: Replica: 6d6fe792107e433f9f0abcda363c876b, State: Running, Role: LEADER
I20260927 10:48:29.399303 27117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } }
I20260927 10:48:29.399849 27113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:29.400907 27119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d6fe792107e433f9f0abcda363c876b. Latest consensus state: current_term: 3 leader_uuid: "6d6fe792107e433f9f0abcda363c876b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } } }
I20260927 10:48:29.401052 27119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:29.400916 27118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6d6fe792107e433f9f0abcda363c876b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d6fe792107e433f9f0abcda363c876b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43695 } } }
I20260927 10:48:29.401211 27118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:29.405441 27132 catalog_manager.cc:1259] Loaded cluster ID: 1428d134ddfe4db885055ebed40fa13b
I20260927 10:48:29.405594 27132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: loading cluster ID for follower catalog manager: success
I20260927 10:48:29.408221 27132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d6fe792107e433f9f0abcda363c876b: acquiring CA information for follower catalog manager: success
F20260927 10:48:29.409484 27132 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa0e78dd9ad google::LogMessage::Fail() at ??:0
@ 0x7fa0e78e088a google::LogMessage::SendToLog() at ??:0
@ 0x7fa0e78dd4de google::LogMessage::Flush() at ??:0
@ 0x7fa0e78dea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa0ea5a118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa0f2242989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fa0f221f023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fa0f221f54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fa0f2218652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fa0f22179e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fa0f225e541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa0e9b20512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa0eb1646db start_thread at ??:0
@ 0x7fa0e63db61f clone at ??:0
W20260927 10:48:29.501351 27115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:29.501691 27115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:29.501739 27115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:29.505846 27115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:29.505904 27115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:29.505964 27115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:29.511379 27115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:44649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=40431
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:29.512516 27115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:29.513679 27115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:29.524431 27153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.524577 27154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.524688 27156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:29.525136 27115 server_base.cc:1034] running on GCE node
I20260927 10:48:29.525671 27115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:29.526321 27115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:29.527504 27115 hybrid_clock.cc:648] HybridClock initialized: now 1790506109527472 us; error 52 us; skew 500 ppm
Sep 27 10:48:29 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:29.530368 27115 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:48:29.531306 27115 webserver.cc:460] Webserver started at http://127.10.96.193:40431/ using document root <none> and password file <none>
I20260927 10:48:29.532004 27115 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:29.532070 27115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:29.536211 27115 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:48:29.538544 27163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:29.539983 27115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:48:29.540136 27115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "cf251b4b08fc4ce59de7feae7a00859a"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "d5c9e4b3e48325ea8aea40733f9d2522"
server_key_iv: "7eab34e6dd5b3dfd49752ad1a03bb204"
server_key_version: "encryptionkey@0"
I20260927 10:48:29.540727 27115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:29.553380 27115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:29.557096 27115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:29.557318 27115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:29.558037 27115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:29.559216 27170 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:48:29.564975 27115 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:48:29.565059 27115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 10:48:29.565150 27115 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:48:29.566892 27115 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:48:29.566962 27115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 10:48:29.567273 27170 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap starting.
I20260927 10:48:29.577589 27115 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:44649
I20260927 10:48:29.577827 27277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:44649 every 8 connection(s)
I20260927 10:48:29.578821 27115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:29.584306 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27115
I20260927 10:48:29.584846 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:39237
--local_ip_for_outbound_sockets=127.10.96.194
--tserver_master_addrs=127.10.96.254:43695
--webserver_port=39613
--webserver_interface=127.10.96.194
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:29.588313 27170 log.cc:824] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:29.598851 27170 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:48:29.599316 27170 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Bootstrap complete.
I20260927 10:48:29.600797 27170 ts_tablet_manager.cc:1397] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.014s
I20260927 10:48:29.604156 27170 raft_consensus.cc:348] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:29.604713 27170 raft_consensus.cc:729] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf251b4b08fc4ce59de7feae7a00859a, State: Initialized, Role: FOLLOWER
I20260927 10:48:29.605237 27170 consensus_queue.cc:260] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:29.606381 27170 ts_tablet_manager.cc:1428] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 10:48:29.616331 27280 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:48:29.581201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:53705)
0927 10:48:29.581843 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.581912 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:29.583117 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:29.615376 (+ 32259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":180,"threads_started":1}
W20260927 10:48:29.616959 27278 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:43695 (0 consecutive failures): Network error: Failed to ping master at 127.10.96.254:43695: Client connection negotiation failed: client connection to 127.10.96.254:43695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 10:48:29.617726 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.617410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:33989)
0927 10:48:29.617599 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.617669 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 10:48:29.618429 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.618226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:41011)
0927 10:48:29.618336 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.618384 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:48:29.618613 27278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 10:48:29.703576 27281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:29.703887 27281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:29.703965 27281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:29.707998 27281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:29.708112 27281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:29.708206 27281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:29.713451 27281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:39237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=39613
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:29.714572 27281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:29.715629 27281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:29.725107 27288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.725136 27291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.725174 27289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:29.725258 27281 server_base.cc:1034] running on GCE node
I20260927 10:48:29.725773 27281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:29.726341 27281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:29.727499 27281 hybrid_clock.cc:648] HybridClock initialized: now 1790506109727477 us; error 34 us; skew 500 ppm
Sep 27 10:48:29 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:29.730525 27281 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:48:29.731309 27281 webserver.cc:460] Webserver started at http://127.10.96.194:39613/ using document root <none> and password file <none>
I20260927 10:48:29.731820 27281 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:29.731879 27281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:29.735837 27281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:29.738045 27298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:29.739342 27281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:29.739445 27281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "e4c70b545b5f4664a0e16e7a950dd586"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "5ba267274247c06317d2b9ab7ad8bdba"
server_key_iv: "f5a6eb2d63e07d5b125175550539c66f"
server_key_version: "encryptionkey@0"
I20260927 10:48:29.740017 27281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:29.757727 27281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:29.760938 27281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:29.761107 27281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:29.761674 27281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:29.762758 27305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 10:48:29.767091 27281 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 10:48:29.767197 27281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 10:48:29.767278 27281 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 10:48:29.769044 27281 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 10:48:29.769094 27281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 10:48:29.769450 27305 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Bootstrap starting.
I20260927 10:48:29.775843 27305 log.cc:824] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:29.779531 27281 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:39237
I20260927 10:48:29.779597 27413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:39237 every 8 connection(s)
I20260927 10:48:29.780431 27281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:29.783373 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27281
I20260927 10:48:29.783826 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:40613
--local_ip_for_outbound_sockets=127.10.96.195
--tserver_master_addrs=127.10.96.254:43695
--webserver_port=36375
--webserver_interface=127.10.96.195
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:48:29.783870 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.782360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:41107)
0927 10:48:29.782857 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.782991 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260927 10:48:29.784356 27305 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260927 10:48:29.784392 27414 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:43695 (0 consecutive failures): Network error: Failed to ping master at 127.10.96.254:43695: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
I20260927 10:48:29.784854 27305 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Bootstrap complete.
I20260927 10:48:29.785216 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.784930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:50017)
0927 10:48:29.785097 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.785152 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 10:48:29.785216 27305 ts_tablet_manager.cc:1397] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260927 10:48:29.785982 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.785709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:39133)
0927 10:48:29.785853 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.785911 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 10:48:29.786247 27414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:48:29.788245 27305 raft_consensus.cc:348] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:29.788676 27305 raft_consensus.cc:729] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c70b545b5f4664a0e16e7a950dd586, State: Initialized, Role: FOLLOWER
I20260927 10:48:29.789162 27305 consensus_queue.cc:260] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:29.790220 27305 ts_tablet_manager.cc:1428] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 10:48:29.790328 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.790031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:44381)
0927 10:48:29.790178 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.790246 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:48:29.791051 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.790774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:34207)
0927 10:48:29.790912 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.790976 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:48:29.897976 27417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:29.898239 27417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:29.898321 27417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:29.902313 27417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:29.902376 27417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:29.902436 27417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:29.907681 27417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:40613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=36375
--tserver_master_addrs=127.10.96.254:43695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:29.908846 27417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:29.909996 27417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:29.919343 27424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.919353 27423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:29.919404 27426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:29.919677 27417 server_base.cc:1034] running on GCE node
I20260927 10:48:29.920064 27417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:29.920655 27417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:29.921819 27417 hybrid_clock.cc:648] HybridClock initialized: now 1790506109921803 us; error 31 us; skew 500 ppm
Sep 27 10:48:29 dist-test-slave-wmg0 krb5kdc[25971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:29.924751 27417 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:48:29.925678 27417 webserver.cc:460] Webserver started at http://127.10.96.195:36375/ using document root <none> and password file <none>
I20260927 10:48:29.926235 27417 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:29.926295 27417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:29.930120 27417 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:29.932189 27433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:29.933503 27417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:29.933660 27417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "dabaf2d0a5d8464894bbd4d8357be138"
format_stamp: "Formatted at 2026-09-27 10:47:47 on dist-test-slave-wmg0"
server_key: "adff109ab8689b9b4886117ca2e20f69"
server_key_iv: "aa2338e419b936ccca94361a7b3d31b7"
server_key_version: "encryptionkey@0"
I20260927 10:48:29.934176 27417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:29.949326 27417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:29.952811 27417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:29.953014 27417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:29.953658 27417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:29.954468 27417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:29.954511 27417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:29.954579 27417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:29.954622 27417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:29.963948 27417 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:40613
I20260927 10:48:29.963985 27546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:40613 every 8 connection(s)
I20260927 10:48:29.964985 27417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:29.968884 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.967206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:38719)
0927 10:48:29.967647 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.967754 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
W20260927 10:48:29.969358 27547 heartbeater.cc:644] Failed to heartbeat to 127.10.96.254:43695 (0 consecutive failures): Network error: Failed to ping master at 127.10.96.254:43695: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
I20260927 10:48:29.969954 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.969750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:49037)
0927 10:48:29.969853 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.969901 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 10:48:29.970561 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:29.970376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:33447)
0927 10:48:29.970461 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:29.970514 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 10:48:29.970744 27547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 10:48:29.970788 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27417
I20260927 10:48:30.619720 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:30.619244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:42803)
0927 10:48:30.619547 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:30.619610 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
I20260927 10:48:30.792233 27552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:30.791834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:39417)
0927 10:48:30.792082 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:30.792142 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260927 10:48:30.971736 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:30.971343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:59343)
0927 10:48:30.971573 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:30.971623 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260927 10:48:31.062350 27554 raft_consensus.cc:482] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:31.062582 27554 raft_consensus.cc:504] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:31.063692 27554 leader_election.cc:288] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e4c70b545b5f4664a0e16e7a950dd586 (127.10.96.194:39237), dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613)
Sep 27 10:48:31 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.195@KRBTEST.COM
Sep 27 10:48:31 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.194@KRBTEST.COM
I20260927 10:48:31.077720 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.063951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:40613 (local address 127.10.96.193:37921)
0927 10:48:31.064487 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.064500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:31.064607 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:31.066108 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:31.066111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:31.066480 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:31.067041 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:31.067056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.068342 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.068347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:31.069516 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:31.069528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.070259 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.070262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:31.070522 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:31.071227 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:31.071240 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:31.072826 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:31.075307 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:31.075313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:31.075327 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:31.075585 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:31.075927 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:31.075931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:31.075933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:31.076083 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:31.076731 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:31.076737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:31.076952 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 10:48:31.077145 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 10:48:31.077527 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":252,"spinlock_wait_cycles":8320,"thread_start_us":75,"threads_started":1}
I20260927 10:48:31.077731 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.064099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:37921 (local address 127.10.96.195:40613)
0927 10:48:31.064494 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:31.064514 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:31.064600 (+ 86us) server_negotiation.cc:407] Connection header received
0927 10:48:31.065435 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:31.065442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:31.065864 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:31.066175 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:31.067212 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.068192 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.069672 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.070125 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.070316 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:31.073098 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:31.073105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:31.073124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:31.073164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:31.075167 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:31.075701 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:31.075706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:31.075712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:31.075832 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:31.076185 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:31.076190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:31.076191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:31.076607 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:31.076746 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:31.077255 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 10:48:31.077539 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":84,"threads_started":1}
I20260927 10:48:31.077780 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.063951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:39237 (local address 127.10.96.193:33821)
0927 10:48:31.064071 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.064087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:31.064205 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:31.065866 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:31.065873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:31.066450 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:31.067040 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:31.067055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.068355 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.068358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:31.069527 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:31.069536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.070265 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.070269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:31.070520 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:31.071204 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:31.071220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:31.073216 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:31.075512 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:31.075517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:31.075519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:31.075717 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:31.076093 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:31.076098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:31.076100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:31.076180 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:31.076797 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:31.076800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:31.076958 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 10:48:31.077144 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 10:48:31.077611 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":442,"spinlock_wait_cycles":30336}
I20260927 10:48:31.078351 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.064096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:33821 (local address 127.10.96.194:39237)
0927 10:48:31.064494 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:31.064511 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:31.064557 (+ 46us) server_negotiation.cc:407] Connection header received
0927 10:48:31.064852 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:31.064858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:31.065378 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:31.066408 (+ 1030us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:31.067263 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.068197 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.069635 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:31.070128 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:31.070311 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:31.073366 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:31.073371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:31.073382 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:31.073409 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:31.075371 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:31.075834 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:31.075840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:31.075847 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:31.075973 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:31.076279 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:31.076284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:31.076286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:31.076687 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:31.076827 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:31.077260 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 10:48:31.077572 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":87,"threads_started":1}
I20260927 10:48:31.079402 27367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed9b114d3a439fab6d9cdd106b8a53" candidate_uuid: "cf251b4b08fc4ce59de7feae7a00859a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e4c70b545b5f4664a0e16e7a950dd586" is_pre_election: true
I20260927 10:48:31.079733 27367 raft_consensus.cc:2454] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf251b4b08fc4ce59de7feae7a00859a in term 1.
W20260927 10:48:31.080461 27165 leader_election.cc:341] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613): Not found: Tablet not found: 88ed9b114d3a439fab6d9cdd106b8a53
I20260927 10:48:31.080662 27166 leader_election.cc:302] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf251b4b08fc4ce59de7feae7a00859a, e4c70b545b5f4664a0e16e7a950dd586; no voters: dabaf2d0a5d8464894bbd4d8357be138
I20260927 10:48:31.080961 27554 raft_consensus.cc:2781] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 10:48:31.081074 27554 raft_consensus.cc:482] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:31.081133 27554 raft_consensus.cc:3037] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:48:31.083343 27554 raft_consensus.cc:504] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:31.083719 27554 leader_election.cc:288] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [CANDIDATE]: Term 2 election: Requested vote from peers e4c70b545b5f4664a0e16e7a950dd586 (127.10.96.194:39237), dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613)
I20260927 10:48:31.084468 27367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed9b114d3a439fab6d9cdd106b8a53" candidate_uuid: "cf251b4b08fc4ce59de7feae7a00859a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e4c70b545b5f4664a0e16e7a950dd586"
W20260927 10:48:31.084579 27165 leader_election.cc:341] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613): Not found: Tablet not found: 88ed9b114d3a439fab6d9cdd106b8a53
I20260927 10:48:31.084633 27367 raft_consensus.cc:3037] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 1 FOLLOWER]: Advancing to term 2
I20260927 10:48:31.086804 27367 raft_consensus.cc:2454] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf251b4b08fc4ce59de7feae7a00859a in term 2.
I20260927 10:48:31.087137 27166 leader_election.cc:302] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf251b4b08fc4ce59de7feae7a00859a, e4c70b545b5f4664a0e16e7a950dd586; no voters: dabaf2d0a5d8464894bbd4d8357be138
I20260927 10:48:31.087306 27554 raft_consensus.cc:2781] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 10:48:31.087473 27554 raft_consensus.cc:686] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [term 2 LEADER]: Becoming Leader. State: Replica: cf251b4b08fc4ce59de7feae7a00859a, State: Running, Role: LEADER
I20260927 10:48:31.087786 27554 consensus_queue.cc:237] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:31.088305 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.088040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:60891)
0927 10:48:31.088168 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.088229 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 10:48:31.565860 27165 consensus_peers.cc:487] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a -> Peer dabaf2d0a5d8464894bbd4d8357be138 (127.10.96.195:40613): Couldn't send request to peer dabaf2d0a5d8464894bbd4d8357be138. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88ed9b114d3a439fab6d9cdd106b8a53. This is attempt 1: this message will repeat every 5th retry.
I20260927 10:48:31.659600 27367 raft_consensus.cc:1261] T 88ed9b114d3a439fab6d9cdd106b8a53 P e4c70b545b5f4664a0e16e7a950dd586 [term 2 FOLLOWER]: Refusing update from remote peer cf251b4b08fc4ce59de7feae7a00859a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 10:48:31.660478 27554 consensus_queue.cc:1035] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 10:48:31.662860 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.662464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:54767)
0927 10:48:31.662684 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.662758 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":57,"threads_started":1}
I20260927 10:48:31.665434 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.665063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:50215)
0927 10:48:31.665261 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.665318 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 10:48:31.665781 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.665521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:37055)
0927 10:48:31.665675 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.665725 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 10:48:31.972812 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:31.972399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:45935)
0927 10:48:31.972642 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:31.972697 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260927 10:48:32.034970 27568 ts_tablet_manager.cc:927] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Initiating tablet copy from peer cf251b4b08fc4ce59de7feae7a00859a (127.10.96.193:44649)
I20260927 10:48:32.035642 27568 tablet_copy_client.cc:316] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: tablet copy: Beginning tablet copy session from remote peer at address 127.10.96.193:44649
Sep 27 10:48:32 dist-test-slave-wmg0 krb5kdc[25971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.193@KRBTEST.COM
I20260927 10:48:32.045974 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.036211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:44649 (local address 127.10.96.195:43529)
0927 10:48:32.036355 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:32.036374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:32.036503 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:32.037136 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:32.037143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:32.037703 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:32.037879 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:32.037886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:32.038919 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:32.038922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:32.039551 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:32.039560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:32.040111 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:32.040113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:32.040191 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:32.040697 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:32.040708 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:32.042097 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:32.044267 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:32.044275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:32.044280 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:32.044499 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:32.044763 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:32.044765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:32.044767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:32.044840 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:32.045383 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:32.045388 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:32.045563 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 10:48:32.045702 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 10:48:32.045837 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 10:48:32.046080 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.036396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:43529 (local address 127.10.96.193:44649)
0927 10:48:32.036692 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:32.036708 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:32.036737 (+ 29us) server_negotiation.cc:407] Connection header received
0927 10:48:32.036783 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:32.036787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:32.036972 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:32.037095 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:32.037995 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:32.038793 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:32.039666 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:32.040006 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:32.040098 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:32.042230 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:32.042239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:32.042251 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:32.042275 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:32.044173 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:32.044595 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:32.044601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:32.044602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:32.044684 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:32.044944 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:32.044947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:32.044948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:32.045277 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:32.045385 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:32.045791 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 10:48:32.045919 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
I20260927 10:48:32.047068 27252 tablet_copy_service.cc:140] P cf251b4b08fc4ce59de7feae7a00859a: Received BeginTabletCopySession request for tablet 88ed9b114d3a439fab6d9cdd106b8a53 from peer dabaf2d0a5d8464894bbd4d8357be138 ({username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:43529)
I20260927 10:48:32.047233 27252 tablet_copy_service.cc:161] P cf251b4b08fc4ce59de7feae7a00859a: Beginning new tablet copy session on tablet 88ed9b114d3a439fab6d9cdd106b8a53 from peer dabaf2d0a5d8464894bbd4d8357be138 at {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:43529: session id = dabaf2d0a5d8464894bbd4d8357be138-88ed9b114d3a439fab6d9cdd106b8a53
I20260927 10:48:32.048339 27252 tablet_copy_source_session.cc:205] T 88ed9b114d3a439fab6d9cdd106b8a53 P cf251b4b08fc4ce59de7feae7a00859a: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 10:48:32.052378 27568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed9b114d3a439fab6d9cdd106b8a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:32.059176 27568 tablet_copy_client.cc:791] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: tablet copy: Starting download of 0 data blocks...
I20260927 10:48:32.059456 27568 tablet_copy_client.cc:655] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: tablet copy: Starting download of 1 WAL segments...
I20260927 10:48:32.061359 27568 tablet_copy_client.cc:531] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 10:48:32.063196 27568 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Bootstrap starting.
I20260927 10:48:32.068552 27568 log.cc:824] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:32.075214 27568 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 10:48:32.075547 27568 tablet_bootstrap.cc:492] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Bootstrap complete.
I20260927 10:48:32.075876 27568 ts_tablet_manager.cc:1397] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260927 10:48:32.078259 27568 raft_consensus.cc:348] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:32.078531 27568 raft_consensus.cc:729] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabaf2d0a5d8464894bbd4d8357be138, State: Initialized, Role: FOLLOWER
I20260927 10:48:32.079016 27568 consensus_queue.cc:260] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4c70b545b5f4664a0e16e7a950dd586" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 39237 } } peers { permanent_uuid: "dabaf2d0a5d8464894bbd4d8357be138" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 40613 } } peers { permanent_uuid: "cf251b4b08fc4ce59de7feae7a00859a" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 44649 } }
I20260927 10:48:32.079970 27568 ts_tablet_manager.cc:1428] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260927 10:48:32.080493 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.080020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:37259)
0927 10:48:32.080324 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:32.080404 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 10:48:32.080868 27252 tablet_copy_service.cc:342] P cf251b4b08fc4ce59de7feae7a00859a: Request end of tablet copy session dabaf2d0a5d8464894bbd4d8357be138-88ed9b114d3a439fab6d9cdd106b8a53 received from {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:43529
I20260927 10:48:32.081007 27252 tablet_copy_service.cc:433] P cf251b4b08fc4ce59de7feae7a00859a: ending tablet copy session dabaf2d0a5d8464894bbd4d8357be138-88ed9b114d3a439fab6d9cdd106b8a53 on tablet 88ed9b114d3a439fab6d9cdd106b8a53 with peer dabaf2d0a5d8464894bbd4d8357be138
I20260927 10:48:32.081331 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.081013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:39097)
0927 10:48:32.081192 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:32.081261 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 10:48:32.612471 27574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.612103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:60917)
0927 10:48:32.612322 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:32.612379 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260927 10:48:32.612417 27501 raft_consensus.cc:1203] T 88ed9b114d3a439fab6d9cdd106b8a53 P dabaf2d0a5d8464894bbd4d8357be138 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 10:48:32.613544 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.613182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:43677)
0927 10:48:32.613351 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:32.613402 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":51,"threads_started":1}
I20260927 10:48:32.666870 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:32.666477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:57653)
0927 10:48:32.666717 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:32.666775 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
I20260927 10:48:33.613826 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:33.613340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:55503)
0927 10:48:33.613625 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:33.613692 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260927 10:48:33.614833 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:33.614271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:35215)
0927 10:48:33.614623 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:33.614712 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260927 10:48:33.668215 27580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:33.667710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:47649)
0927 10:48:33.668026 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:33.668118 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":87,"threads_started":1}
I20260927 10:48:34.615274 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:34.614776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:49167)
0927 10:48:34.615050 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:34.615155 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260927 10:48:34.615947 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:34.615579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:38697)
0927 10:48:34.615812 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:34.615861 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 10:48:34.616031 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:34.615797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:37015)
0927 10:48:34.615916 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:34.615977 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:48:34.669654 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:34.669073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:35425)
0927 10:48:34.669424 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:34.669545 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":91,"threads_started":1}
I20260927 10:48:35.617322 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:35.616902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:46617)
0927 10:48:35.617170 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:35.617227 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260927 10:48:35.617389 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:35.616999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:50753)
0927 10:48:35.617244 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:35.617304 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260927 10:48:35.670993 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:35.670595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:38921)
0927 10:48:35.670841 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:35.670899 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260927 10:48:36.618713 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:36.618229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:40255)
0927 10:48:36.618527 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:36.618594 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":123,"threads_started":1}
I20260927 10:48:36.618757 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:36.618229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:54277)
0927 10:48:36.618614 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:36.618665 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":124,"threads_started":1}
I20260927 10:48:36.672261 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:36.671775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:53187)
0927 10:48:36.672032 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:36.672092 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260927 10:48:37.619925 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:37.619504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:35153)
0927 10:48:37.619768 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:37.619827 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260927 10:48:37.619982 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:37.619504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:37373)
0927 10:48:37.619841 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:37.619893 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":92,"threads_started":1}
I20260927 10:48:37.673530 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:37.673064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:60733)
0927 10:48:37.673321 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:37.673377 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260927 10:48:38.621341 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:38.620924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:59427)
0927 10:48:38.621189 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:38.621247 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260927 10:48:38.621423 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:38.620924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:42383)
0927 10:48:38.621282 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:38.621338 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260927 10:48:38.674711 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:38.674276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:34341)
0927 10:48:38.674539 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:38.674609 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260927 10:48:39.622761 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:39.622318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:44091)
0927 10:48:39.622595 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:39.622659 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":95,"threads_started":1}
I20260927 10:48:39.622822 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:39.622345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:48059)
0927 10:48:39.622677 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:39.622735 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":75,"threads_started":1}
I20260927 10:48:39.675881 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:39.675457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:42395)
0927 10:48:39.675722 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:39.675780 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":106,"threads_started":1}
I20260927 10:48:40.624238 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:40.623776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:45871)
0927 10:48:40.624070 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:40.624138 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260927 10:48:40.624330 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:40.623776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:38869)
0927 10:48:40.624157 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:40.624214 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":95,"threads_started":1}
I20260927 10:48:40.677233 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:40.676829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:40223)
0927 10:48:40.677083 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:40.677142 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260927 10:48:41.625545 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:41.625026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:56207)
0927 10:48:41.625314 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:41.625376 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260927 10:48:41.625630 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:41.625197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:45505)
0927 10:48:41.625434 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:41.625546 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260927 10:48:41.678593 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:41.678152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:58559)
0927 10:48:41.678424 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:41.678494 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260927 10:48:42.626817 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:42.626399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:54267)
0927 10:48:42.626661 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:42.626723 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260927 10:48:42.626847 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:42.626393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:56609)
0927 10:48:42.626717 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:42.626765 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":100,"threads_started":1}
I20260927 10:48:42.679857 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:42.679419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:40963)
0927 10:48:42.679701 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:42.679761 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260927 10:48:43.628173 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:43.627707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:40353)
0927 10:48:43.627987 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:43.628076 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
I20260927 10:48:43.628181 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:43.627707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:46273)
0927 10:48:43.628047 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:43.628098 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260927 10:48:43.681154 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:43.680718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:48553)
0927 10:48:43.680991 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:43.681060 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260927 10:48:44.629647 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:44.629018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.195:35225)
0927 10:48:44.629405 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:44.629515 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260927 10:48:44.630000 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:44.629102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.193:32975)
0927 10:48:44.629448 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:44.629887 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":58,"threads_started":1}
I20260927 10:48:44.682534 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:44.682080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43695 (local address 127.10.96.194:41933)
0927 10:48:44.682369 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:44.682433 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.254:43695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260927 10:48:44.980814 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27115
I20260927 10:48:44.986691 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27281
I20260927 10:48:44.992476 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27417
2026-09-27T10:48:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59351 ms)
[----------] 30 tests from SecurityITest (240551 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:45 dist-test-slave-wmg0 krb5kdc[27617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:45 dist-test-slave-wmg0 krb5kdc[27617](info): set up 2 sockets
Sep 27 10:48:45 dist-test-slave-wmg0 krb5kdc[27617](info): commencing operation
krb5kdc: starting...
W20260927 10:48:46.037322 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:48:46.162587 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43155
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:46.273051 27630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:46.273407 27630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:46.273516 27630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:46.277680 27630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:46.277793 27630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:46.277827 27630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:46.277863 27630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:46.277894 27630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:46.283330 27630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:46.284499 27630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:46.285601 27630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:48:46.294392 27630 server_base.cc:1034] running on GCE node
W20260927 10:48:46.294560 27638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:46.294601 27636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:46.294750 27635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:46.295364 27630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:46.296542 27630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:46.297690 27630 hybrid_clock.cc:648] HybridClock initialized: now 1790506126297660 us; error 45 us; skew 500 ppm
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:46.300520 27630 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:48:46.301450 27630 webserver.cc:460] Webserver started at http://127.10.96.254:37035/ using document root <none> and password file <none>
I20260927 10:48:46.302044 27630 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:46.302102 27630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:46.302268 27630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:46.304307 27630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "d1859ed3a8494eefa70e9c910781bf63"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "42a2c8cd0458ad823aebb456dd90bf50"
server_key_iv: "7324722a91036b35659124fdf11f3e9a"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.304833 27630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "d1859ed3a8494eefa70e9c910781bf63"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "42a2c8cd0458ad823aebb456dd90bf50"
server_key_iv: "7324722a91036b35659124fdf11f3e9a"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.308609 27630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 10:48:46.311105 27645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.312373 27630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:48:46.312475 27630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "d1859ed3a8494eefa70e9c910781bf63"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "42a2c8cd0458ad823aebb456dd90bf50"
server_key_iv: "7324722a91036b35659124fdf11f3e9a"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.312582 27630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:46.323746 27630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:46.327108 27630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:46.327267 27630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:46.334489 27630 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43155
I20260927 10:48:46.334517 27697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43155 every 8 connection(s)
I20260927 10:48:46.335440 27630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:46.338188 27698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:46.339094 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27630
I20260927 10:48:46.339234 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:46.339534 10627 external_mini_cluster.cc:1428] Setting key 6888e2e72e7287a810c19e7cf7ba957a
I20260927 10:48:46.344182 27698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63: Bootstrap starting.
I20260927 10:48:46.346359 27698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:46.347213 27698 log.cc:824] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:46.349418 27698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63: No bootstrap required, opened a new log
I20260927 10:48:46.353060 27698 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1859ed3a8494eefa70e9c910781bf63" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43155 } }
I20260927 10:48:46.353322 27698 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:46.353394 27698 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1859ed3a8494eefa70e9c910781bf63, State: Initialized, Role: FOLLOWER
I20260927 10:48:46.353869 27698 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1859ed3a8494eefa70e9c910781bf63" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43155 } }
I20260927 10:48:46.353992 27698 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:46.354076 27698 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:46.354171 27698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:46.354168 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:46.340985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.10.96.254:43155)
0927 10:48:46.341429 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:46.341443 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:46.341553 (+ 110us) server_negotiation.cc:407] Connection header received
0927 10:48:46.342185 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:46.342203 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:46.342547 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:46.342882 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:46.343694 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.344698 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.345604 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.346066 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.346238 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:46.349195 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:46.349200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:46.349213 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:46.349239 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:46.351108 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:46.351580 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:46.351591 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:46.351596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:46.351714 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:46.352015 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:46.352018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:46.352020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:46.352404 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:46.352623 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:46.352890 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 10:48:46.353209 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
I20260927 10:48:46.355096 27698 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1859ed3a8494eefa70e9c910781bf63" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43155 } }
I20260927 10:48:46.355468 27698 leader_election.cc:302] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1859ed3a8494eefa70e9c910781bf63; no voters:
I20260927 10:48:46.355813 27698 leader_election.cc:288] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:46.355942 27703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:46.356180 27703 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [term 1 LEADER]: Becoming Leader. State: Replica: d1859ed3a8494eefa70e9c910781bf63, State: Running, Role: LEADER
I20260927 10:48:46.356490 27703 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1859ed3a8494eefa70e9c910781bf63" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43155 } }
I20260927 10:48:46.357096 27698 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:46.357690 27704 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1859ed3a8494eefa70e9c910781bf63. Latest consensus state: current_term: 1 leader_uuid: "d1859ed3a8494eefa70e9c910781bf63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1859ed3a8494eefa70e9c910781bf63" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43155 } } }
I20260927 10:48:46.357913 27704 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:46.358144 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1859ed3a8494eefa70e9c910781bf63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1859ed3a8494eefa70e9c910781bf63" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43155 } } }
I20260927 10:48:46.358247 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:46.358364 27712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:46.361580 27712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:46.367475 27712 catalog_manager.cc:1347] Generated new cluster ID: 403dec5765874008b0be0af583b20cb2
I20260927 10:48:46.367554 27712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:46.387920 27712 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:46.389235 27712 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:46.407750 27712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63: Generated new TSK 0
I20260927 10:48:46.408769 27712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:48:46.446367 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43155
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:46.557966 27724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:46.558318 27724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:46.558400 27724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:46.562526 27724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:46.562623 27724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:46.562717 27724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:46.568068 27724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:46.569209 27724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:46.570335 27724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:46.580343 27730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:46.580343 27732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:46.580348 27729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:46.580590 27724 server_base.cc:1034] running on GCE node
I20260927 10:48:46.581168 27724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:46.581884 27724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:46.583074 27724 hybrid_clock.cc:648] HybridClock initialized: now 1790506126583039 us; error 48 us; skew 500 ppm
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:46.586047 27724 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:48:46.586969 27724 webserver.cc:460] Webserver started at http://127.10.96.193:41319/ using document root <none> and password file <none>
I20260927 10:48:46.587553 27724 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:46.587648 27724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:46.587880 27724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:46.590054 27724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "eaa526d0a708463c8fde44d5b804fcfd"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "c8b45cb878bfa2993a9f86a2c8192d12"
server_key_iv: "386ad0fef205265282880b991760eb5c"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.590629 27724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "eaa526d0a708463c8fde44d5b804fcfd"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "c8b45cb878bfa2993a9f86a2c8192d12"
server_key_iv: "386ad0fef205265282880b991760eb5c"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.594594 27724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:46.597033 27739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.598436 27724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:46.598580 27724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "eaa526d0a708463c8fde44d5b804fcfd"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "c8b45cb878bfa2993a9f86a2c8192d12"
server_key_iv: "386ad0fef205265282880b991760eb5c"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.598698 27724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:46.640111 27724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:46.643483 27724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:46.643683 27724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:46.644279 27724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:46.645112 27724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:46.645183 27724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.645262 27724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:46.645313 27724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.653930 27724 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:43309
I20260927 10:48:46.653985 27852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:43309 every 8 connection(s)
I20260927 10:48:46.654850 27724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:46.663666 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27724
I20260927 10:48:46.663825 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:48:46.664216 10627 external_mini_cluster.cc:1428] Setting key e29e7692529588b310b5ac88e2330738
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:46.669314 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:46.656769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:56195 (local address 127.10.96.254:43155)
0927 10:48:46.656915 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:46.656919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:46.657783 (+ 864us) server_negotiation.cc:407] Connection header received
0927 10:48:46.659226 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:46.659230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:46.659290 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:46.659404 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:46.660573 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.661105 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.662268 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.662509 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.662629 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:46.665274 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:46.665278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:46.665281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:46.665314 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:46.666841 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:46.667515 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:46.667521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:46.667523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:46.667647 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:46.668019 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:46.668024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:46.668028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:46.668274 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:46.668390 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:46.668923 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 10:48:46.669066 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:48:46.670104 27855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:46.657146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43155 (local address 127.10.96.193:56195)
0927 10:48:46.657628 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:46.657673 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:46.658997 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:46.659579 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:46.659586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:46.659921 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:46.660410 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:46.660423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.661228 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.661232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:46.662139 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:46.662148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.662664 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.662668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:46.662837 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:46.663395 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:46.663411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:46.665106 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:46.666960 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:46.666969 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:46.666988 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:46.667371 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:46.667768 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:46.667771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:46.667773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:46.667911 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:46.668422 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:46.668427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:46.668676 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 10:48:46.668837 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 10:48:46.669120 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":98,"threads_started":1}
I20260927 10:48:46.671404 27853 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43155
I20260927 10:48:46.671669 27853 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:46.672142 27853 heartbeater.cc:511] Master 127.10.96.254:43155 requested a full tablet report, sending...
I20260927 10:48:46.673795 27662 ts_manager.cc:194] Registered new tserver with Master: eaa526d0a708463c8fde44d5b804fcfd (127.10.96.193:43309)
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:48:46.675071 27662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:56195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:48:46.690757 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43155
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:46.802201 27858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:46.802495 27858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:46.802539 27858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:46.806773 27858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:46.806839 27858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:46.806942 27858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:46.812512 27858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:46.813668 27858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:46.814801 27858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:46.824672 27863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:46.824717 27866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:46.824668 27864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:46.824983 27858 server_base.cc:1034] running on GCE node
I20260927 10:48:46.825400 27858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:46.826046 27858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:46.827229 27858 hybrid_clock.cc:648] HybridClock initialized: now 1790506126827195 us; error 44 us; skew 500 ppm
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:46.830093 27858 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:48:46.830967 27858 webserver.cc:460] Webserver started at http://127.10.96.194:46483/ using document root <none> and password file <none>
I20260927 10:48:46.831528 27858 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:46.831624 27858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:46.831848 27858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:46.833942 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "43188b57c1844c46b6d9c7faf92d7df0"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "c0357a46b92036c3bfedce9b419a7ec8"
server_key_iv: "92aa3fdb5b969bf53ca62213ebbcd0ce"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.834501 27858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "43188b57c1844c46b6d9c7faf92d7df0"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "c0357a46b92036c3bfedce9b419a7ec8"
server_key_iv: "92aa3fdb5b969bf53ca62213ebbcd0ce"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.838249 27858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:46.840623 27873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.841970 27858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:46.842110 27858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "43188b57c1844c46b6d9c7faf92d7df0"
format_stamp: "Formatted at 2026-09-27 10:48:46 on dist-test-slave-wmg0"
server_key: "c0357a46b92036c3bfedce9b419a7ec8"
server_key_iv: "92aa3fdb5b969bf53ca62213ebbcd0ce"
server_key_version: "encryptionkey@0"
I20260927 10:48:46.842222 27858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:46.887199 27858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:46.890329 27858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:46.890537 27858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:46.891255 27858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:46.892089 27858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:46.892160 27858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.892231 27858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:46.892282 27858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:46.901238 27858 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:45969
I20260927 10:48:46.901260 27986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:45969 every 8 connection(s)
I20260927 10:48:46.902330 27858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:46.907703 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27858
I20260927 10:48:46.907802 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:48:46.908159 10627 external_mini_cluster.cc:1428] Setting key ea1f506c930a1ce995c7e4b16bb054e2
Sep 27 10:48:46 dist-test-slave-wmg0 krb5kdc[27617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506126, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:46.917340 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:46.904148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:35915 (local address 127.10.96.254:43155)
0927 10:48:46.904279 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:46.904283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:46.905040 (+ 757us) server_negotiation.cc:407] Connection header received
0927 10:48:46.906492 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:46.906496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:46.906578 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:46.906683 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:46.908119 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.908835 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.910392 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.910620 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.910710 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:46.913623 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:46.913644 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:46.913649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:46.913682 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:46.915214 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:46.915797 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:46.915814 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:46.915815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:46.915888 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:46.916252 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:46.916255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:46.916257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:46.916439 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:46.916564 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:46.917056 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 10:48:46.917176 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:48:46.918289 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:46.904465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43155 (local address 127.10.96.194:35915)
0927 10:48:46.904888 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:46.904931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:46.906278 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:46.906855 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:46.906862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:46.907236 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:46.907897 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:46.907920 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.908943 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.908946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:46.910268 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:46.910283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:46.910772 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:46.910778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:46.911014 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:46.911716 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:46.911731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:46.913403 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:46.915332 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:46.915338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:46.915354 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:46.915688 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:46.915999 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:46.916002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:46.916004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:46.916151 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:46.916543 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:46.916550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:46.916817 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 10:48:46.917002 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 10:48:46.917362 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260927 10:48:46.919524 27987 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43155
I20260927 10:48:46.919803 27987 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
I20260927 10:48:46.920336 27987 heartbeater.cc:511] Master 127.10.96.254:43155 requested a full tablet report, sending...
I20260927 10:48:46.921618 27662 ts_manager.cc:194] Registered new tserver with Master: 43188b57c1844c46b6d9c7faf92d7df0 (127.10.96.194:45969)
I20260927 10:48:46.922282 27662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:35915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:48:46.936611 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43155
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:36853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:47.047672 27992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:47.047931 27992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:47.047974 27992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:47.052151 27992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:47.052212 27992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:47.052297 27992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:47.057852 27992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:36853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:43155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:47.059130 27992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:47.060176 27992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:47.069788 28000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.069782 27997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.069782 27998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:47.070297 27992 server_base.cc:1034] running on GCE node
I20260927 10:48:47.070715 27992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:47.071278 27992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:47.072443 27992 hybrid_clock.cc:648] HybridClock initialized: now 1790506127072404 us; error 47 us; skew 500 ppm
Sep 27 10:48:47 dist-test-slave-wmg0 krb5kdc[27617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506127, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:47.075237 27992 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:48:47.076161 27992 webserver.cc:460] Webserver started at http://127.10.96.195:36689/ using document root <none> and password file <none>
I20260927 10:48:47.076728 27992 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:47.076817 27992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:47.077045 27992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:47.079133 27992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "0c4bb0f70daf45c4828896ed13455587"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "342375e96900a12ef502f0de15e6cecf"
server_key_iv: "5545fae46641512b32aaf32f8191e4fc"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.079746 27992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "0c4bb0f70daf45c4828896ed13455587"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "342375e96900a12ef502f0de15e6cecf"
server_key_iv: "5545fae46641512b32aaf32f8191e4fc"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.083629 27992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 10:48:47.085949 28007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.087195 27992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:47.087317 27992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "0c4bb0f70daf45c4828896ed13455587"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "342375e96900a12ef502f0de15e6cecf"
server_key_iv: "5545fae46641512b32aaf32f8191e4fc"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.087421 27992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:47.100395 27992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:47.103649 27992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:47.103832 27992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:47.104393 27992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:47.105173 27992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:47.105218 27992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.105258 27992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:47.105273 27992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.113971 27992 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:39849
I20260927 10:48:47.114005 28120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:39849 every 8 connection(s)
I20260927 10:48:47.114879 27992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:47.123258 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 27992
I20260927 10:48:47.123411 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:48:47.123720 10627 external_mini_cluster.cc:1428] Setting key 1e095fc3432a8b04df28daf43fcce4e5
Sep 27 10:48:47 dist-test-slave-wmg0 krb5kdc[27617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506127, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:47.129292 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.116799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:47759 (local address 127.10.96.254:43155)
0927 10:48:47.116932 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.116936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.117872 (+ 936us) server_negotiation.cc:407] Connection header received
0927 10:48:47.119227 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.119231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.119289 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.119390 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:47.120541 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.121081 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.122272 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.122492 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.122614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.125530 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:47.125537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:47.125540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:47.125576 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:47.127169 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:47.127713 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:47.127717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:47.127718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:47.127800 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:47.128164 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:47.128168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:47.128170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:47.128381 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:47.128473 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.129044 (+ 571us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.129162 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:48:47.130191 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.117245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:43155 (local address 127.10.96.195:47759)
0927 10:48:47.117724 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.117765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.119008 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.119566 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.119574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.119917 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:47.120390 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.120403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.121222 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.121225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.122143 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.122152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.122654 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.122658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.122830 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.123503 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:47.123526 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:47.125349 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:47.127293 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:47.127298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:47.127311 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:47.127609 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:47.127893 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:47.127895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:47.127897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:47.128047 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:47.128524 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:47.128530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:47.128790 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.128958 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.129245 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":163,"threads_started":1}
I20260927 10:48:47.131383 28121 heartbeater.cc:347] Connected to a master server at 127.10.96.254:43155
I20260927 10:48:47.131670 28121 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:47.132205 28121 heartbeater.cc:511] Master 127.10.96.254:43155 requested a full tablet report, sending...
I20260927 10:48:47.133365 27662 ts_manager.cc:194] Registered new tserver with Master: 0c4bb0f70daf45c4828896ed13455587 (127.10.96.195:39849)
I20260927 10:48:47.134016 27662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:47759
I20260927 10:48:47.137893 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:47.147661 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.139891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.10.96.254:43155)
0927 10:48:47.140029 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.140035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.140217 (+ 182us) server_negotiation.cc:407] Connection header received
0927 10:48:47.140372 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.140375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.140424 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.140509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:47.141205 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.141807 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.142659 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.142876 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.142986 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.144448 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:47.144454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:47.144456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:47.144491 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:47.145952 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:47.146389 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:47.146394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:47.146396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:47.146479 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:47.146755 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:47.146760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:47.146762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:47.146993 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:47.147138 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.147374 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.147490 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:48:47.150666 27662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:47.153875 27662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:47.169090 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.162518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:45969 (local address 127.0.0.1:52266)
0927 10:48:47.162909 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.162963 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.163089 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.164408 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.164415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.164450 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:47.164613 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.164621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.166196 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.166199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.167418 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.167427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.168499 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.168505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.168632 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.168684 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.168789 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.168915 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":50,"thread_start_us":87,"threads_started":1}
I20260927 10:48:47.169137 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.162518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:39849 (local address 127.0.0.1:47204)
0927 10:48:47.163462 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.163473 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.163566 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.164883 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.164886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.164903 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:47.164976 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.164981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.165986 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.165992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.167681 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.167690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.168725 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.168728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.168813 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.168834 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.168914 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.168999 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":96,"thread_start_us":95,"threads_started":1}
I20260927 10:48:47.169508 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.163436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.10.96.193:43309)
0927 10:48:47.163755 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.163774 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.164074 (+ 300us) server_negotiation.cc:407] Connection header received
0927 10:48:47.164221 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.164226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.164410 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.164558 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:47.164799 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.166055 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.167938 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.168580 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.168682 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.168817 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.169215 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.169323 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260927 10:48:47.169565 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.163239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43309 (local address 127.0.0.1:59282)
0927 10:48:47.163967 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.163979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.164096 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.164548 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.164551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.164570 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:47.164680 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.164686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.166692 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.166695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.167814 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.167830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.168876 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.168879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.168956 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.168976 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.169056 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.169126 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":92,"threads_started":1}
I20260927 10:48:47.169150 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.162661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.10.96.195:39849)
0927 10:48:47.164360 (+ 1699us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.164372 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.164396 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:48:47.164479 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.164501 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.164699 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.164905 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:47.165079 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.165861 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.167806 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.168615 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.168712 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.168838 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.168942 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.169031 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":164,"threads_started":1}
I20260927 10:48:47.169854 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.163540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.10.96.194:45969)
0927 10:48:47.163867 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.163879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.163904 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:48:47.163982 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.163988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.164232 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.164776 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:47.164932 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.166055 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.167540 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.168360 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.168475 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.168626 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.169450 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.169591 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20260927 10:48:47.172284 27787 tablet_service.cc:1467] Processing CreateTablet for tablet c9f5af08bc6345e7bc87f53bc755a396 (DEFAULT_TABLE table=test-table [id=30013668b9fe48bf935681b8d7ca3b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:47.172286 28055 tablet_service.cc:1467] Processing CreateTablet for tablet c9f5af08bc6345e7bc87f53bc755a396 (DEFAULT_TABLE table=test-table [id=30013668b9fe48bf935681b8d7ca3b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:47.173223 28055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f5af08bc6345e7bc87f53bc755a396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:47.173223 27787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f5af08bc6345e7bc87f53bc755a396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:47.179772 28138 tablet_bootstrap.cc:492] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Bootstrap starting.
I20260927 10:48:47.179490 27921 tablet_service.cc:1467] Processing CreateTablet for tablet c9f5af08bc6345e7bc87f53bc755a396 (DEFAULT_TABLE table=test-table [id=30013668b9fe48bf935681b8d7ca3b8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:47.180351 28139 tablet_bootstrap.cc:492] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd: Bootstrap starting.
I20260927 10:48:47.180581 27921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9f5af08bc6345e7bc87f53bc755a396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:47.181926 28138 tablet_bootstrap.cc:654] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:47.182356 28139 tablet_bootstrap.cc:654] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:47.183075 28138 log.cc:824] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:47.183182 28139 log.cc:824] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:47.185205 28138 tablet_bootstrap.cc:492] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: No bootstrap required, opened a new log
I20260927 10:48:47.185406 28138 ts_tablet_manager.cc:1397] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:48:47.185701 28139 tablet_bootstrap.cc:492] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd: No bootstrap required, opened a new log
I20260927 10:48:47.185886 28139 ts_tablet_manager.cc:1397] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:48:47.187667 28142 tablet_bootstrap.cc:492] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: Bootstrap starting.
I20260927 10:48:47.189087 28138 raft_consensus.cc:348] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.189332 28138 raft_consensus.cc:374] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:47.189383 28138 raft_consensus.cc:729] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4bb0f70daf45c4828896ed13455587, State: Initialized, Role: FOLLOWER
I20260927 10:48:47.189852 28138 consensus_queue.cc:260] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.189951 28142 tablet_bootstrap.cc:654] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:47.189785 28139 raft_consensus.cc:348] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.190073 28139 raft_consensus.cc:374] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:47.190124 28139 raft_consensus.cc:729] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa526d0a708463c8fde44d5b804fcfd, State: Initialized, Role: FOLLOWER
I20260927 10:48:47.190624 28139 consensus_queue.cc:260] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.190896 28138 ts_tablet_manager.cc:1428] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260927 10:48:47.191272 27853 heartbeater.cc:503] Master 127.10.96.254:43155 was elected leader, sending a full tablet report...
I20260927 10:48:47.191272 28121 heartbeater.cc:503] Master 127.10.96.254:43155 was elected leader, sending a full tablet report...
I20260927 10:48:47.191578 28139 ts_tablet_manager.cc:1428] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:47.192004 28142 log.cc:824] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:47.193990 28142 tablet_bootstrap.cc:492] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: No bootstrap required, opened a new log
I20260927 10:48:47.194214 28142 ts_tablet_manager.cc:1397] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 10:48:47.197731 28142 raft_consensus.cc:348] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.197974 28142 raft_consensus.cc:374] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:47.198068 28142 raft_consensus.cc:729] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43188b57c1844c46b6d9c7faf92d7df0, State: Initialized, Role: FOLLOWER
I20260927 10:48:47.198478 28142 consensus_queue.cc:260] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.199128 27987 heartbeater.cc:503] Master 127.10.96.254:43155 was elected leader, sending a full tablet report...
I20260927 10:48:47.199328 28142 ts_tablet_manager.cc:1428] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 10:48:47.286932 28143 raft_consensus.cc:482] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:47.287140 28143 raft_consensus.cc:504] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.288234 28143 leader_election.cc:288] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43188b57c1844c46b6d9c7faf92d7df0 (127.10.96.194:45969), eaa526d0a708463c8fde44d5b804fcfd (127.10.96.193:43309)
I20260927 10:48:47.292598 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.288632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:35381 (local address 127.10.96.194:45969)
0927 10:48:47.288756 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.288759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.288774 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:48:47.288912 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.288918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.288973 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.289085 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:47.289368 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.289944 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.291312 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.291963 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.292123 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.292176 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.292357 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.292462 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:48:47.292589 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.288527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:45969 (local address 127.10.96.195:35381)
0927 10:48:47.288662 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.288681 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.288784 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.289097 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.289101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.289123 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:47.289247 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.289260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.290055 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.290062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.291170 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.291180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.292112 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.292116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.292226 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.292253 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.292337 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.292428 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 10:48:47.292845 28147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.288857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43309 (local address 127.10.96.195:37361)
0927 10:48:47.289241 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.289255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.289368 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.289664 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.289667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.289684 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:47.289816 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.289821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.290606 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.290610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.291589 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.291596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.292430 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.292434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.292543 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.292566 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.292649 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.292721 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":161,"threads_started":1}
I20260927 10:48:47.292922 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.289023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:37361 (local address 127.10.96.193:43309)
0927 10:48:47.289239 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.289243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.289344 (+ 101us) server_negotiation.cc:407] Connection header received
0927 10:48:47.289515 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.289518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.289572 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.289655 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:47.289935 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.290468 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.291771 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.292316 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.292428 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.292472 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.292681 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.292766 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 10:48:47.293602 27807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f5af08bc6345e7bc87f53bc755a396" candidate_uuid: "0c4bb0f70daf45c4828896ed13455587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa526d0a708463c8fde44d5b804fcfd" is_pre_election: true
I20260927 10:48:47.293599 27941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f5af08bc6345e7bc87f53bc755a396" candidate_uuid: "0c4bb0f70daf45c4828896ed13455587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43188b57c1844c46b6d9c7faf92d7df0" is_pre_election: true
I20260927 10:48:47.293908 27807 raft_consensus.cc:2454] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4bb0f70daf45c4828896ed13455587 in term 0.
I20260927 10:48:47.293908 27941 raft_consensus.cc:2454] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c4bb0f70daf45c4828896ed13455587 in term 0.
I20260927 10:48:47.294392 28010 leader_election.cc:302] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4bb0f70daf45c4828896ed13455587, eaa526d0a708463c8fde44d5b804fcfd; no voters:
I20260927 10:48:47.294704 28143 raft_consensus.cc:2781] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:47.294785 28143 raft_consensus.cc:482] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:47.294816 28143 raft_consensus.cc:3037] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:47.296098 28143 raft_consensus.cc:504] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.296454 28143 leader_election.cc:288] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [CANDIDATE]: Term 1 election: Requested vote from peers 43188b57c1844c46b6d9c7faf92d7df0 (127.10.96.194:45969), eaa526d0a708463c8fde44d5b804fcfd (127.10.96.193:43309)
I20260927 10:48:47.296866 27941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f5af08bc6345e7bc87f53bc755a396" candidate_uuid: "0c4bb0f70daf45c4828896ed13455587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43188b57c1844c46b6d9c7faf92d7df0"
I20260927 10:48:47.297000 27807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9f5af08bc6345e7bc87f53bc755a396" candidate_uuid: "0c4bb0f70daf45c4828896ed13455587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaa526d0a708463c8fde44d5b804fcfd"
I20260927 10:48:47.296999 27941 raft_consensus.cc:3037] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:47.297183 27807 raft_consensus.cc:3037] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:47.298249 27941 raft_consensus.cc:2454] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4bb0f70daf45c4828896ed13455587 in term 1.
I20260927 10:48:47.298341 27807 raft_consensus.cc:2454] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c4bb0f70daf45c4828896ed13455587 in term 1.
I20260927 10:48:47.298573 28010 leader_election.cc:302] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c4bb0f70daf45c4828896ed13455587, 43188b57c1844c46b6d9c7faf92d7df0; no voters:
I20260927 10:48:47.298789 28143 raft_consensus.cc:2781] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:47.299046 28143 raft_consensus.cc:686] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 1 LEADER]: Becoming Leader. State: Replica: 0c4bb0f70daf45c4828896ed13455587, State: Running, Role: LEADER
I20260927 10:48:47.299384 28143 consensus_queue.cc:237] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } }
I20260927 10:48:47.303022 27661 catalog_manager.cc:5495] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c4bb0f70daf45c4828896ed13455587 (127.10.96.195). New cstate: current_term: 1 leader_uuid: "0c4bb0f70daf45c4828896ed13455587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c4bb0f70daf45c4828896ed13455587" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 39849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260927 10:48:47.322425 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.318743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.10.96.195:39849)
0927 10:48:47.318890 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.318895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.318909 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:48:47.319054 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.319056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.319114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.319224 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:47.319484 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.319995 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.320852 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.321081 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.321189 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.321562 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:47.321995 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:47.322132 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.322201 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.322257 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:48:47.328451 27807 raft_consensus.cc:1261] T c9f5af08bc6345e7bc87f53bc755a396 P eaa526d0a708463c8fde44d5b804fcfd [term 1 FOLLOWER]: Refusing update from remote peer 0c4bb0f70daf45c4828896ed13455587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:47.328509 27941 raft_consensus.cc:1261] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 1 FOLLOWER]: Refusing update from remote peer 0c4bb0f70daf45c4828896ed13455587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:47.329191 28143 consensus_queue.cc:1035] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaa526d0a708463c8fde44d5b804fcfd" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:47.329452 28148 consensus_queue.cc:1035] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43188b57c1844c46b6d9c7faf92d7df0" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:47.337329 28151 mvcc.cc:205] Tried to move back new op lower bound from 7333913097529602048 to 7333913097421086720. Current Snapshot: MvccSnapshot[applied={T|T < 7333913097529602048}]
I20260927 10:48:47.338788 28153 mvcc.cc:205] Tried to move back new op lower bound from 7333913097529602048 to 7333913097421086720. Current Snapshot: MvccSnapshot[applied={T|T < 7333913097529602048}]
I20260927 10:48:47.340579 28152 mvcc.cc:205] Tried to move back new op lower bound from 7333913097529602048 to 7333913097421086720. Current Snapshot: MvccSnapshot[applied={T|T < 7333913097529602048}]
I20260927 10:48:47.345703 27661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:47.345881 27661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:47.348265 27661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1859ed3a8494eefa70e9c910781bf63: Sending DeleteTablet for 3 replicas of tablet c9f5af08bc6345e7bc87f53bc755a396
I20260927 10:48:47.348974 28055 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f5af08bc6345e7bc87f53bc755a396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:47 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:47204
I20260927 10:48:47.349154 27921 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f5af08bc6345e7bc87f53bc755a396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:47 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:52266
I20260927 10:48:47.349210 27787 tablet_service.cc:1514] Processing DeleteTablet for tablet c9f5af08bc6345e7bc87f53bc755a396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:47 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:59282
I20260927 10:48:47.350309 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27724
I20260927 10:48:47.353636 28161 tablet_replica.cc:330] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: stopping tablet replica
I20260927 10:48:47.353956 28161 raft_consensus.cc:2229] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:48:47.354395 28161 raft_consensus.cc:2258] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:47.354784 28162 tablet_replica.cc:330] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: stopping tablet replica
I20260927 10:48:47.355072 28162 raft_consensus.cc:2229] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:47.355365 28162 raft_consensus.cc:2258] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:47.356981 28161 ts_tablet_manager.cc:1905] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:47.357013 28162 ts_tablet_manager.cc:1905] T c9f5af08bc6345e7bc87f53bc755a396 P 43188b57c1844c46b6d9c7faf92d7df0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:47.359879 28161 ts_tablet_manager.cc:1918] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:47.359983 28161 log.cc:1196] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/c9f5af08bc6345e7bc87f53bc755a396
I20260927 10:48:47.360313 28161 ts_tablet_manager.cc:1939] T c9f5af08bc6345e7bc87f53bc755a396 P 0c4bb0f70daf45c4828896ed13455587: Deleting consensus metadata
W20260927 10:48:47.361220 27648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:43309 (error 108)
I20260927 10:48:47.361433 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27858
I20260927 10:48:47.361608 27647 catalog_manager.cc:4841] TS 0c4bb0f70daf45c4828896ed13455587 (127.10.96.195:39849): tablet c9f5af08bc6345e7bc87f53bc755a396 (table test-table [id=30013668b9fe48bf935681b8d7ca3b8e]) successfully deleted
I20260927 10:48:47.362890 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.362575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:43309 (local address 127.0.0.1:59294)
0927 10:48:47.362727 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.362803 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:43309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 10:48:47.363054 27648 catalog_manager.cc:4584] TS eaa526d0a708463c8fde44d5b804fcfd (127.10.96.193:43309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9f5af08bc6345e7bc87f53bc755a396: Network error: Client connection negotiation failed: client connection to 127.10.96.193:43309: connect: Connection refused (error 111)
I20260927 10:48:47.368259 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27992
I20260927 10:48:47.369241 28133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.368467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:45969 (local address 127.0.0.1:52282)
0927 10:48:47.369055 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.369136 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486}
I20260927 10:48:47.374017 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 27630
2026-09-27T10:48:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2381 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T10:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:47Z Disabled control of system clock
I20260927 10:48:47.404958 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:42863
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:42863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:48:47.514652 28168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:47.514940 28168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:47.514976 28168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:47.519053 28168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:47.519119 28168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:47.519134 28168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:47.519151 28168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:47.519217 28168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:47.524108 28168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:42863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:42863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:47.525233 28168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:47.526657 28168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:47.534832 28173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.534899 28176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.534839 28174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:47.534987 28168 server_base.cc:1034] running on GCE node
I20260927 10:48:47.535610 28168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:47.536657 28168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:47.537813 28168 hybrid_clock.cc:648] HybridClock initialized: now 1790506127537797 us; error 31 us; skew 500 ppm
I20260927 10:48:47.539740 28168 webserver.cc:460] Webserver started at http://127.10.96.254:41707/ using document root <none> and password file <none>
I20260927 10:48:47.540328 28168 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:47.540388 28168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:47.540607 28168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:47.542755 28168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "b9f06de24de54884a0cc9f5872a65477"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "bcaea5e50a6fe49c6ee412674db491e6"
server_key_iv: "6abfcf48c2d15ff0125fd5a8274bad90"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.543322 28168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "b9f06de24de54884a0cc9f5872a65477"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "bcaea5e50a6fe49c6ee412674db491e6"
server_key_iv: "6abfcf48c2d15ff0125fd5a8274bad90"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.547021 28168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:48:47.549383 28182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.550676 28168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:47.550812 28168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "b9f06de24de54884a0cc9f5872a65477"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "bcaea5e50a6fe49c6ee412674db491e6"
server_key_iv: "6abfcf48c2d15ff0125fd5a8274bad90"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.550925 28168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:47.577992 28168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:47.578738 28168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:47.578943 28168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:47.585896 28168 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:42863
I20260927 10:48:47.585923 28234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:42863 every 8 connection(s)
I20260927 10:48:47.586838 28168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:47.590211 28235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:47.590904 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28168
I20260927 10:48:47.591045 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:47.591317 10627 external_mini_cluster.cc:1428] Setting key 96848fcf2045ceb644ce384d679ebbcc
I20260927 10:48:47.596458 28235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477: Bootstrap starting.
I20260927 10:48:47.598794 28235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:47.599542 28235 log.cc:824] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:47.600999 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.592711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.10.96.254:42863)
0927 10:48:47.593393 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.593408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.593444 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:48:47.594185 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.594203 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.594609 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.594956 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:47.595399 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.596546 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.597581 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.598065 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.598242 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.599154 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:47.599159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:47.599172 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:47.599567 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:47.599623 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:47.599643 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:47.599775 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.599927 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.600221 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":130,"threads_started":1}
I20260927 10:48:47.601722 28235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477: No bootstrap required, opened a new log
I20260927 10:48:47.605141 28235 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f06de24de54884a0cc9f5872a65477" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42863 } }
I20260927 10:48:47.605376 28235 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:47.605458 28235 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f06de24de54884a0cc9f5872a65477, State: Initialized, Role: FOLLOWER
I20260927 10:48:47.605890 28235 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f06de24de54884a0cc9f5872a65477" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42863 } }
I20260927 10:48:47.606053 28235 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:47.606133 28235 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:47.606236 28235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:47.607323 28235 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f06de24de54884a0cc9f5872a65477" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42863 } }
I20260927 10:48:47.607651 28235 leader_election.cc:302] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9f06de24de54884a0cc9f5872a65477; no voters:
I20260927 10:48:47.607955 28235 leader_election.cc:288] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:47.608070 28240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:47.608316 28240 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [term 1 LEADER]: Becoming Leader. State: Replica: b9f06de24de54884a0cc9f5872a65477, State: Running, Role: LEADER
I20260927 10:48:47.608601 28240 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f06de24de54884a0cc9f5872a65477" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42863 } }
I20260927 10:48:47.609146 28235 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:47.610378 28241 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9f06de24de54884a0cc9f5872a65477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f06de24de54884a0cc9f5872a65477" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42863 } } }
I20260927 10:48:47.610386 28242 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9f06de24de54884a0cc9f5872a65477. Latest consensus state: current_term: 1 leader_uuid: "b9f06de24de54884a0cc9f5872a65477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9f06de24de54884a0cc9f5872a65477" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42863 } } }
I20260927 10:48:47.610545 28241 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:47.610551 28242 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:47.610883 28249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:47.614003 28249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:47.619565 28249 catalog_manager.cc:1347] Generated new cluster ID: f91f7545dd8d4774906ac0f18be8c058
I20260927 10:48:47.619635 28249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:47.641606 28249 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:47.642846 28249 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:47.650574 28249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477: Generated new TSK 0
I20260927 10:48:47.651361 28249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:47.663182 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42863
--builtin_ntp_servers=127.10.96.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:48:47.776233 28259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:47.776546 28259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:47.776616 28259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:47.780807 28259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:47.780893 28259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:47.780989 28259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:47.785846 28259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:47.786973 28259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:47.788401 28259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:47.798080 28267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.798026 28264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:47.798061 28259 server_base.cc:1034] running on GCE node
W20260927 10:48:47.798031 28265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:47.798663 28259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:47.799310 28259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:47.800472 28259 hybrid_clock.cc:648] HybridClock initialized: now 1790506127800434 us; error 51 us; skew 500 ppm
I20260927 10:48:47.802331 28259 webserver.cc:460] Webserver started at http://127.10.96.193:33269/ using document root <none> and password file <none>
I20260927 10:48:47.802897 28259 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:47.802959 28259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:47.803165 28259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:47.805176 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "822802bbab5a4503bc0d94f27d707ff6"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "d9353f9f23711318ae85f3f70181877e"
server_key_iv: "89abe560ca2d29e38431fc3c235436d4"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.805783 28259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "822802bbab5a4503bc0d94f27d707ff6"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "d9353f9f23711318ae85f3f70181877e"
server_key_iv: "89abe560ca2d29e38431fc3c235436d4"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.809446 28259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 10:48:47.812042 28273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.813266 28259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 10:48:47.813387 28259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "822802bbab5a4503bc0d94f27d707ff6"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "d9353f9f23711318ae85f3f70181877e"
server_key_iv: "89abe560ca2d29e38431fc3c235436d4"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.813552 28259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:47.822819 28259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:47.823482 28259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:47.823683 28259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:47.824244 28259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:47.825148 28259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:47.825196 28259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.825259 28259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:47.825289 28259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.834450 28259 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:32979
I20260927 10:48:47.834470 28386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:32979 every 8 connection(s)
I20260927 10:48:47.835444 28259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:47.839591 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28259
I20260927 10:48:47.839715 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:48:47.839998 10627 external_mini_cluster.cc:1428] Setting key f31f15b5095b393284afd9dd2babad54
I20260927 10:48:47.842038 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42863
--builtin_ntp_servers=127.10.96.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:48:47.845284 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.837369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:37783 (local address 127.10.96.254:42863)
0927 10:48:47.837575 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:47.837579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:47.838664 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 10:48:47.840247 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:47.840251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:47.840318 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:47.840440 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:47.841580 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.842105 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.843319 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.843549 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.843661 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.844813 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:47.844816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:47.844818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:47.844830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:47.844851 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:47.844859 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:47.844923 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:47.845112 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:48:47.845173 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:48:47.846592 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:47.837832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42863 (local address 127.10.96.193:37783)
0927 10:48:47.838506 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:47.838548 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:47.839952 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:47.840657 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:47.840666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:47.840780 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:47.841386 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.841427 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.842292 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.842296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.843191 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:47.843199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:47.843717 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:47.843721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:47.843889 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:47.844500 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:47.844504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:47.844561 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:47.844645 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:47.844947 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:47.844958 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:47.844967 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:48:47.845105 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:48:47.845412 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260927 10:48:47.847922 28387 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42863
I20260927 10:48:47.848206 28387 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:47.848896 28387 heartbeater.cc:511] Master 127.10.96.254:42863 requested a full tablet report, sending...
I20260927 10:48:47.850824 28199 ts_manager.cc:194] Registered new tserver with Master: 822802bbab5a4503bc0d94f27d707ff6 (127.10.96.193:32979)
I20260927 10:48:47.852614 28199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.193:37783
W20260927 10:48:47.955188 28390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:47.955456 28390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:47.955502 28390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:47.959625 28390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:47.959724 28390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:47.959785 28390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:47.964535 28390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:47.965627 28390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:47.966970 28390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:47.976830 28395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.976835 28398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:47.976974 28396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:47.977087 28390 server_base.cc:1034] running on GCE node
I20260927 10:48:47.977660 28390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:47.978387 28390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:47.979609 28390 hybrid_clock.cc:648] HybridClock initialized: now 1790506127979593 us; error 31 us; skew 500 ppm
I20260927 10:48:47.981434 28390 webserver.cc:460] Webserver started at http://127.10.96.194:36269/ using document root <none> and password file <none>
I20260927 10:48:47.982057 28390 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:47.982160 28390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:47.982399 28390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:47.984437 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "ee996ae449c9490991bbfffdfbd0915f"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "5d592591d73bd6274d90b45fb2469f96"
server_key_iv: "68c04e4868a7b8a3e33b9d5c1d1ab4a7"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.985019 28390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "ee996ae449c9490991bbfffdfbd0915f"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "5d592591d73bd6274d90b45fb2469f96"
server_key_iv: "68c04e4868a7b8a3e33b9d5c1d1ab4a7"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.988768 28390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.003s
I20260927 10:48:47.991235 28404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:47.992525 28390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:47.992662 28390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "ee996ae449c9490991bbfffdfbd0915f"
format_stamp: "Formatted at 2026-09-27 10:48:47 on dist-test-slave-wmg0"
server_key: "5d592591d73bd6274d90b45fb2469f96"
server_key_iv: "68c04e4868a7b8a3e33b9d5c1d1ab4a7"
server_key_version: "encryptionkey@0"
I20260927 10:48:47.992779 28390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:48.021799 28390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:48.022631 28390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:48.022858 28390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:48.023505 28390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:48.024578 28390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:48.024650 28390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:48.024724 28390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:48.024766 28390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:48.033862 28390 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:35209
I20260927 10:48:48.033885 28517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:35209 every 8 connection(s)
I20260927 10:48:48.034816 28390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:48.039054 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28390
I20260927 10:48:48.039222 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:48:48.039578 10627 external_mini_cluster.cc:1428] Setting key 77730fbbfd11fc0d67ba9e75986cb5bc
I20260927 10:48:48.041806 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42863
--builtin_ntp_servers=127.10.96.212:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:48:48.044879 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.036740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36303 (local address 127.10.96.254:42863)
0927 10:48:48.036878 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.036881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.037999 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 10:48:48.039493 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.039497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.039564 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.039674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:48.040735 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.041396 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.042757 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.042981 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.043232 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.044373 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:48.044377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:48.044380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:48.044392 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:48.044414 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:48.044435 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:48.044511 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.044679 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.044742 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:48:48.045836 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.037123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42863 (local address 127.10.96.194:36303)
0927 10:48:48.037820 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.037862 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.039161 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.039865 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.039873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.040017 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:48.040590 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.040605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.041611 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.041614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.042620 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.042630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.043267 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.043272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.043454 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.044104 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:48.044105 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:48.044148 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:48.044213 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:48.044524 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:48.044535 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:48.044541 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.044679 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.044991 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260927 10:48:48.047015 28518 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42863
I20260927 10:48:48.047271 28518 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:48.047803 28518 heartbeater.cc:511] Master 127.10.96.254:42863 requested a full tablet report, sending...
I20260927 10:48:48.048902 28199 ts_manager.cc:194] Registered new tserver with Master: ee996ae449c9490991bbfffdfbd0915f (127.10.96.194:35209)
I20260927 10:48:48.049831 28199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.194:36303
W20260927 10:48:48.157018 28521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:48.157294 28521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:48.157334 28521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:48.161365 28521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:48.161432 28521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:48.161551 28521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:48.166395 28521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:48.167462 28521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:48.168803 28521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:48.178431 28526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:48.178414 28527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:48.178517 28521 server_base.cc:1034] running on GCE node
W20260927 10:48:48.178435 28529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:48.179049 28521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:48.179648 28521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:48.180811 28521 hybrid_clock.cc:648] HybridClock initialized: now 1790506128180797 us; error 25 us; skew 500 ppm
I20260927 10:48:48.182827 28521 webserver.cc:460] Webserver started at http://127.10.96.195:41953/ using document root <none> and password file <none>
I20260927 10:48:48.183401 28521 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:48.183463 28521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:48.183683 28521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:48.185801 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "b3435c714a3344e58d959a5d4010d383"
format_stamp: "Formatted at 2026-09-27 10:48:48 on dist-test-slave-wmg0"
server_key: "67a1e214f2990aafd3ff083fedc87b10"
server_key_iv: "7216d753dcfc90794476a40955befc8b"
server_key_version: "encryptionkey@0"
I20260927 10:48:48.186384 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "b3435c714a3344e58d959a5d4010d383"
format_stamp: "Formatted at 2026-09-27 10:48:48 on dist-test-slave-wmg0"
server_key: "67a1e214f2990aafd3ff083fedc87b10"
server_key_iv: "7216d753dcfc90794476a40955befc8b"
server_key_version: "encryptionkey@0"
I20260927 10:48:48.190601 28521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:48.193145 28535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:48.194469 28521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 10:48:48.194625 28521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "b3435c714a3344e58d959a5d4010d383"
format_stamp: "Formatted at 2026-09-27 10:48:48 on dist-test-slave-wmg0"
server_key: "67a1e214f2990aafd3ff083fedc87b10"
server_key_iv: "7216d753dcfc90794476a40955befc8b"
server_key_version: "encryptionkey@0"
I20260927 10:48:48.194751 28521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:48.206517 28521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:48.207233 28521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:48.207433 28521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:48.208015 28521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:48.208971 28521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:48.209050 28521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:48.209124 28521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:48.209172 28521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:48.218230 28521 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:43515
I20260927 10:48:48.218276 28648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:43515 every 8 connection(s)
I20260927 10:48:48.219208 28521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:48.228245 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28521
I20260927 10:48:48.228368 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:48:48.228686 10627 external_mini_cluster.cc:1428] Setting key 4d8bc83ed8b32085f9d52215c7e2513a
I20260927 10:48:48.228691 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.221130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:59803 (local address 127.10.96.254:42863)
0927 10:48:48.221310 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.221315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.222478 (+ 1163us) server_negotiation.cc:407] Connection header received
0927 10:48:48.223885 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.223888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.223939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.224048 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:48.224958 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.225596 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.226725 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.226932 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.227010 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.228133 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:48.228136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:48.228138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:48.228151 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:48.228172 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:48.228179 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:48.228239 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.228498 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.228575 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 10:48:48.229768 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.221664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42863 (local address 127.10.96.195:59803)
0927 10:48:48.222323 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.222364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.223659 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.224192 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.224198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.224285 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:48.224813 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.224826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.225711 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.225713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.226621 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.226630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.227061 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.227063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.227216 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.227813 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:48.227815 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:48.227856 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:48.227946 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:48.228268 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:48.228279 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:48.228285 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.228470 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.228791 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260927 10:48:48.231055 28649 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42863
I20260927 10:48:48.231315 28649 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:48.231824 28649 heartbeater.cc:511] Master 127.10.96.254:42863 requested a full tablet report, sending...
I20260927 10:48:48.232843 28199 ts_manager.cc:194] Registered new tserver with Master: b3435c714a3344e58d959a5d4010d383 (127.10.96.195:43515)
I20260927 10:48:48.233378 28199 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.195:59803
I20260927 10:48:48.243173 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:48.249250 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.245259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.10.96.254:42863)
0927 10:48:48.245411 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.245415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.245638 (+ 223us) server_negotiation.cc:407] Connection header received
0927 10:48:48.245807 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.245812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.245860 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.245946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:48.246241 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.246805 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.247655 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.247878 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.247989 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.248828 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:48.248831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:48.248833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:48.248844 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:48.248864 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:48.248872 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:48.248951 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.249038 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.249097 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:48:48.252321 28199 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:48.255633 28199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:48.270203 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.264928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.10.96.195:43515)
0927 10:48:48.265369 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.265382 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.265408 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:48:48.265520 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.265525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.265749 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.265943 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:48.266281 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.267448 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.268829 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.269453 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.269631 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.269838 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.269932 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.270030 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":135,"threads_started":1}
I20260927 10:48:48.270339 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.264209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:35209 (local address 127.0.0.1:47664)
0927 10:48:48.264922 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.264978 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.265115 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.265941 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.265946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.265982 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:48.266171 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.266180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.267292 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.267297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.268581 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.268591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.269865 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.269869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.269959 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.269985 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.270043 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.270140 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":29,"spinlock_wait_cycles":7936,"thread_start_us":82,"threads_started":1}
I20260927 10:48:48.270203 28662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.264477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:43515 (local address 127.0.0.1:57114)
0927 10:48:48.264922 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.264978 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.265115 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.265943 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.265948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.265978 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:48.266169 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.266175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.267583 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.267586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.268698 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.268709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.269648 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.269651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.269755 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.269792 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.269905 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.270029 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":31,"spinlock_wait_cycles":10880,"thread_start_us":99,"threads_started":1}
I20260927 10:48:48.270823 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.264384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.10.96.194:35209)
0927 10:48:48.265407 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.265419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.265443 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:48:48.265547 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.265552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.265782 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.266304 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:48.266443 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.267173 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.268713 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.269697 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.270271 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.270470 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.270525 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.270642 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":159,"threads_started":1}
I20260927 10:48:48.271128 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.264209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:32979 (local address 127.0.0.1:41026)
0927 10:48:48.265565 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.265578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.265663 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.266287 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.266289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.266305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:48.266378 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.266383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.267708 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.267711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.268873 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.268881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.270655 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.270659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.270816 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.270850 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.270935 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.271008 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"mutex_wait_us":917,"thread_start_us":87,"threads_started":1}
I20260927 10:48:48.271656 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.264436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.10.96.193:32979)
0927 10:48:48.265407 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.265419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.265751 (+ 332us) server_negotiation.cc:407] Connection header received
0927 10:48:48.265825 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.265830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.265994 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.266145 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:48.266502 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.267590 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.268993 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.270057 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.270153 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.271341 (+ 1188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.271394 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.271498 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":154,"threads_started":1}
I20260927 10:48:48.273967 28452 tablet_service.cc:1467] Processing CreateTablet for tablet ad6e67e8fda74223b18caaae8cca2ba8 (DEFAULT_TABLE table=test-table [id=6e96c2df52d64db2a9c87bd690cb743f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:48.273970 28583 tablet_service.cc:1467] Processing CreateTablet for tablet ad6e67e8fda74223b18caaae8cca2ba8 (DEFAULT_TABLE table=test-table [id=6e96c2df52d64db2a9c87bd690cb743f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:48.274101 28321 tablet_service.cc:1467] Processing CreateTablet for tablet ad6e67e8fda74223b18caaae8cca2ba8 (DEFAULT_TABLE table=test-table [id=6e96c2df52d64db2a9c87bd690cb743f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:48.274966 28583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6e67e8fda74223b18caaae8cca2ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:48.274966 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6e67e8fda74223b18caaae8cca2ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:48.274951 28452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6e67e8fda74223b18caaae8cca2ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:48.281438 28666 tablet_bootstrap.cc:492] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Bootstrap starting.
I20260927 10:48:48.283042 28668 tablet_bootstrap.cc:492] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: Bootstrap starting.
I20260927 10:48:48.283821 28666 tablet_bootstrap.cc:654] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:48.284569 28666 log.cc:824] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:48.284847 28668 tablet_bootstrap.cc:654] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:48.285730 28668 log.cc:824] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:48.285847 28667 tablet_bootstrap.cc:492] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Bootstrap starting.
I20260927 10:48:48.287104 28666 tablet_bootstrap.cc:492] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: No bootstrap required, opened a new log
I20260927 10:48:48.287324 28666 ts_tablet_manager.cc:1397] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 10:48:48.287783 28668 tablet_bootstrap.cc:492] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: No bootstrap required, opened a new log
I20260927 10:48:48.287986 28668 ts_tablet_manager.cc:1397] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:48:48.287994 28667 tablet_bootstrap.cc:654] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:48.288878 28667 log.cc:824] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:48.291117 28666 raft_consensus.cc:348] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.291381 28666 raft_consensus.cc:374] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:48.291437 28666 raft_consensus.cc:729] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee996ae449c9490991bbfffdfbd0915f, State: Initialized, Role: FOLLOWER
I20260927 10:48:48.291479 28667 tablet_bootstrap.cc:492] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: No bootstrap required, opened a new log
I20260927 10:48:48.291699 28667 ts_tablet_manager.cc:1397] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:48:48.291956 28666 consensus_queue.cc:260] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.292682 28518 heartbeater.cc:503] Master 127.10.96.254:42863 was elected leader, sending a full tablet report...
I20260927 10:48:48.292913 28668 raft_consensus.cc:348] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.293205 28668 raft_consensus.cc:374] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:48.293263 28668 raft_consensus.cc:729] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822802bbab5a4503bc0d94f27d707ff6, State: Initialized, Role: FOLLOWER
I20260927 10:48:48.293721 28668 consensus_queue.cc:260] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.294493 28666 ts_tablet_manager.cc:1428] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:48:48.294726 28668 ts_tablet_manager.cc:1428] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260927 10:48:48.295125 28387 heartbeater.cc:503] Master 127.10.96.254:42863 was elected leader, sending a full tablet report...
I20260927 10:48:48.296949 28667 raft_consensus.cc:348] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.297259 28667 raft_consensus.cc:374] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:48.297328 28667 raft_consensus.cc:729] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3435c714a3344e58d959a5d4010d383, State: Initialized, Role: FOLLOWER
I20260927 10:48:48.297901 28667 consensus_queue.cc:260] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.298630 28649 heartbeater.cc:503] Master 127.10.96.254:42863 was elected leader, sending a full tablet report...
I20260927 10:48:48.298942 28667 ts_tablet_manager.cc:1428] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260927 10:48:48.336648 28388 tablet.cc:2369] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:48.400583 28672 raft_consensus.cc:482] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:48.400801 28672 raft_consensus.cc:504] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.402050 28672 leader_election.cc:288] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 822802bbab5a4503bc0d94f27d707ff6 (127.10.96.193:32979), b3435c714a3344e58d959a5d4010d383 (127.10.96.195:43515)
I20260927 10:48:48.405675 28520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.402247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:32979 (local address 127.10.96.194:58261)
0927 10:48:48.402370 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.402387 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.402488 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.402831 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.402834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.402852 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:48.402970 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.402976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.403740 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.403745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.404555 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.404564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.405215 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.405219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.405322 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.405340 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.405428 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.405544 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260927 10:48:48.405675 28663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.402445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:58261 (local address 127.10.96.193:32979)
0927 10:48:48.402590 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.402595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.402612 (+ 17us) server_negotiation.cc:407] Connection header received
0927 10:48:48.402675 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.402677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.402717 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.402819 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:48.403113 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.403624 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.404684 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.405090 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.405220 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.405273 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.405444 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.405551 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:48:48.405926 28664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.402449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:60121 (local address 127.10.96.195:43515)
0927 10:48:48.402593 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.402599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.402845 (+ 246us) server_negotiation.cc:407] Connection header received
0927 10:48:48.402991 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.402994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.403049 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.403135 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:48.403398 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.403901 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.404917 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.405311 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.405437 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.405521 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.405710 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.405796 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:48:48.406241 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.402340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:43515 (local address 127.10.96.194:60121)
0927 10:48:48.402745 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.402756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:48.402872 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:48.403154 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:48.403157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:48.403172 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:48.403290 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.403295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.404021 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.404025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.404788 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:48.404796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.405437 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.405441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:48.405585 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.405604 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:48:48.405693 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:48:48.406083 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":122,"threads_started":1}
I20260927 10:48:48.406721 28341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6e67e8fda74223b18caaae8cca2ba8" candidate_uuid: "ee996ae449c9490991bbfffdfbd0915f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822802bbab5a4503bc0d94f27d707ff6" is_pre_election: true
I20260927 10:48:48.406833 28603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6e67e8fda74223b18caaae8cca2ba8" candidate_uuid: "ee996ae449c9490991bbfffdfbd0915f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3435c714a3344e58d959a5d4010d383" is_pre_election: true
I20260927 10:48:48.407073 28341 raft_consensus.cc:2454] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee996ae449c9490991bbfffdfbd0915f in term 0.
I20260927 10:48:48.407073 28603 raft_consensus.cc:2454] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee996ae449c9490991bbfffdfbd0915f in term 0.
I20260927 10:48:48.407577 28406 leader_election.cc:302] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 822802bbab5a4503bc0d94f27d707ff6, ee996ae449c9490991bbfffdfbd0915f; no voters:
I20260927 10:48:48.407828 28672 raft_consensus.cc:2781] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:48.407922 28672 raft_consensus.cc:482] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:48.407975 28672 raft_consensus.cc:3037] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:48.409101 28672 raft_consensus.cc:504] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.409490 28672 leader_election.cc:288] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [CANDIDATE]: Term 1 election: Requested vote from peers 822802bbab5a4503bc0d94f27d707ff6 (127.10.96.193:32979), b3435c714a3344e58d959a5d4010d383 (127.10.96.195:43515)
I20260927 10:48:48.409824 28341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6e67e8fda74223b18caaae8cca2ba8" candidate_uuid: "ee996ae449c9490991bbfffdfbd0915f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "822802bbab5a4503bc0d94f27d707ff6"
I20260927 10:48:48.409942 28603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad6e67e8fda74223b18caaae8cca2ba8" candidate_uuid: "ee996ae449c9490991bbfffdfbd0915f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3435c714a3344e58d959a5d4010d383"
I20260927 10:48:48.409982 28341 raft_consensus.cc:3037] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:48.410085 28603 raft_consensus.cc:3037] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:48.411329 28341 raft_consensus.cc:2454] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee996ae449c9490991bbfffdfbd0915f in term 1.
I20260927 10:48:48.411329 28603 raft_consensus.cc:2454] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee996ae449c9490991bbfffdfbd0915f in term 1.
I20260927 10:48:48.411687 28405 leader_election.cc:302] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 822802bbab5a4503bc0d94f27d707ff6, b3435c714a3344e58d959a5d4010d383, ee996ae449c9490991bbfffdfbd0915f; no voters:
I20260927 10:48:48.411945 28672 raft_consensus.cc:2781] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:48.412194 28672 raft_consensus.cc:686] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 1 LEADER]: Becoming Leader. State: Replica: ee996ae449c9490991bbfffdfbd0915f, State: Running, Role: LEADER
I20260927 10:48:48.412533 28672 consensus_queue.cc:237] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } }
I20260927 10:48:48.415987 28198 catalog_manager.cc:5495] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee996ae449c9490991bbfffdfbd0915f (127.10.96.194). New cstate: current_term: 1 leader_uuid: "ee996ae449c9490991bbfffdfbd0915f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee996ae449c9490991bbfffdfbd0915f" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 35209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 } health_report { overall_health: UNKNOWN } } }
I20260927 10:48:48.461436 28661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.457802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.10.96.194:35209)
0927 10:48:48.457972 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:48.457979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:48.457995 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:48:48.458091 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:48.458093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:48.458148 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:48.458264 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:48.458510 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.459043 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.459945 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:48.460171 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:48.460286 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:48.460591 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:48.461029 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:48.461171 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:48.461218 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 10:48:48.461268 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 10:48:48.467229 28603 raft_consensus.cc:1261] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 1 FOLLOWER]: Refusing update from remote peer ee996ae449c9490991bbfffdfbd0915f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:48.467228 28341 raft_consensus.cc:1261] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 1 FOLLOWER]: Refusing update from remote peer ee996ae449c9490991bbfffdfbd0915f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:48.467979 28672 consensus_queue.cc:1035] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [LEADER]: Connected to new peer: Peer: permanent_uuid: "822802bbab5a4503bc0d94f27d707ff6" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:48.468207 28676 consensus_queue.cc:1035] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3435c714a3344e58d959a5d4010d383" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:48:48.470124 28650 tablet.cc:2369] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:48.475740 28679 mvcc.cc:205] Tried to move back new op lower bound from 7333913102194556928 to 7333913101980258304. Current Snapshot: MvccSnapshot[applied={T|T < 7333913102194556928}]
I20260927 10:48:48.484282 28199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:48.484429 28199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:48.486750 28199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9f06de24de54884a0cc9f5872a65477: Sending DeleteTablet for 3 replicas of tablet ad6e67e8fda74223b18caaae8cca2ba8
I20260927 10:48:48.487534 28583 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6e67e8fda74223b18caaae8cca2ba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:48 UTC) from {username='slave'} at 127.0.0.1:57114
I20260927 10:48:48.487542 28452 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6e67e8fda74223b18caaae8cca2ba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:48 UTC) from {username='slave'} at 127.0.0.1:47664
I20260927 10:48:48.487553 28321 tablet_service.cc:1514] Processing DeleteTablet for tablet ad6e67e8fda74223b18caaae8cca2ba8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:48 UTC) from {username='slave'} at 127.0.0.1:41026
I20260927 10:48:48.488104 28691 tablet_replica.cc:330] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6: stopping tablet replica
I20260927 10:48:48.488245 28690 tablet_replica.cc:330] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: stopping tablet replica
I20260927 10:48:48.488435 28691 raft_consensus.cc:2229] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:48.488643 28690 raft_consensus.cc:2229] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:48:48.488775 28691 raft_consensus.cc:2258] T ad6e67e8fda74223b18caaae8cca2ba8 P 822802bbab5a4503bc0d94f27d707ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:48.488878 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28259
I20260927 10:48:48.489161 28690 raft_consensus.cc:2258] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:48.489706 28689 tablet_replica.cc:330] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: stopping tablet replica
I20260927 10:48:48.490057 28689 raft_consensus.cc:2229] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:48.490391 28689 raft_consensus.cc:2258] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:48.493026 28690 ts_tablet_manager.cc:1905] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:48.495328 28690 ts_tablet_manager.cc:1918] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:48.495440 28690 log.cc:1196] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/ad6e67e8fda74223b18caaae8cca2ba8
I20260927 10:48:48.495829 28690 ts_tablet_manager.cc:1939] T ad6e67e8fda74223b18caaae8cca2ba8 P ee996ae449c9490991bbfffdfbd0915f: Deleting consensus metadata
W20260927 10:48:48.496542 28184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:32979 (error 108)
I20260927 10:48:48.496603 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28390
I20260927 10:48:48.497094 28689 ts_tablet_manager.cc:1905] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:48.499290 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.498938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:32979 (local address 127.0.0.1:41032)
0927 10:48:48.499094 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.499156 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 10:48:48.499502 28184 catalog_manager.cc:4584] TS 822802bbab5a4503bc0d94f27d707ff6 (127.10.96.193:32979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad6e67e8fda74223b18caaae8cca2ba8: Network error: Client connection negotiation failed: client connection to 127.10.96.193:32979: connect: Connection refused (error 111)
I20260927 10:48:48.500694 28689 ts_tablet_manager.cc:1918] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:48.500806 28689 log.cc:1196] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/ad6e67e8fda74223b18caaae8cca2ba8
I20260927 10:48:48.501120 28689 ts_tablet_manager.cc:1939] T ad6e67e8fda74223b18caaae8cca2ba8 P b3435c714a3344e58d959a5d4010d383: Deleting consensus metadata
I20260927 10:48:48.502605 28183 catalog_manager.cc:4841] TS b3435c714a3344e58d959a5d4010d383 (127.10.96.195:43515): tablet ad6e67e8fda74223b18caaae8cca2ba8 (table test-table [id=6e96c2df52d64db2a9c87bd690cb743f]) successfully deleted
I20260927 10:48:48.503963 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28521
I20260927 10:48:48.504877 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:48.504038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:35209 (local address 127.0.0.1:47672)
0927 10:48:48.504762 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:48.504807 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:35209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":651}
I20260927 10:48:48.509636 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28168
2026-09-27T10:48:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:48 dist-test-slave-wmg0 krb5kdc[28695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:48 dist-test-slave-wmg0 krb5kdc[28695](info): set up 2 sockets
Sep 27 10:48:48 dist-test-slave-wmg0 krb5kdc[28695](info): commencing operation
krb5kdc: starting...
W20260927 10:48:49.541549 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:48:49 dist-test-slave-wmg0 krb5kdc[28695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:48:49.666046 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:32975
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:49.778388 28708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:49.778827 28708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:49.778910 28708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:49.783641 28708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:49.783725 28708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:49.783763 28708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:49.783802 28708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:49.783836 28708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:49.789324 28708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:49.790625 28708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:49.791716 28708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:49.800474 28713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:49.800462 28714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:49.800498 28716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:49.800904 28708 server_base.cc:1034] running on GCE node
I20260927 10:48:49.801604 28708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:49.802740 28708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:49.803933 28708 hybrid_clock.cc:648] HybridClock initialized: now 1790506129803915 us; error 28 us; skew 500 ppm
Sep 27 10:48:49 dist-test-slave-wmg0 krb5kdc[28695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506129, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:49.806905 28708 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:48:49.807901 28708 webserver.cc:460] Webserver started at http://127.10.96.254:38623/ using document root <none> and password file <none>
I20260927 10:48:49.808482 28708 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:49.808542 28708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:49.808789 28708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:49.810925 28708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "d45c577faefd4f61b509a3eecf11bb94"
format_stamp: "Formatted at 2026-09-27 10:48:49 on dist-test-slave-wmg0"
server_key: "b6e3f68baea3a984c51c784bfe372f10"
server_key_iv: "3b9242b256f0fe2887e58b34781f8cbb"
server_key_version: "encryptionkey@0"
I20260927 10:48:49.811498 28708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "d45c577faefd4f61b509a3eecf11bb94"
format_stamp: "Formatted at 2026-09-27 10:48:49 on dist-test-slave-wmg0"
server_key: "b6e3f68baea3a984c51c784bfe372f10"
server_key_iv: "3b9242b256f0fe2887e58b34781f8cbb"
server_key_version: "encryptionkey@0"
I20260927 10:48:49.815299 28708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 10:48:49.817739 28723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:49.819001 28708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:48:49.819142 28708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "d45c577faefd4f61b509a3eecf11bb94"
format_stamp: "Formatted at 2026-09-27 10:48:49 on dist-test-slave-wmg0"
server_key: "b6e3f68baea3a984c51c784bfe372f10"
server_key_iv: "3b9242b256f0fe2887e58b34781f8cbb"
server_key_version: "encryptionkey@0"
I20260927 10:48:49.819257 28708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:49.848753 28708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:49.852471 28708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:49.852711 28708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:49.859992 28708 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:32975
I20260927 10:48:49.860034 28775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:32975 every 8 connection(s)
I20260927 10:48:49.861076 28708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:49.862807 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28708
I20260927 10:48:49.862958 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:49.863282 10627 external_mini_cluster.cc:1428] Setting key 9cc9dca1848983aeef365261d41d053a
I20260927 10:48:49.864591 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:49.870292 28776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94: Bootstrap starting.
Sep 27 10:48:49 dist-test-slave-wmg0 krb5kdc[28695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:49.872720 28776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:49.873442 28776 log.cc:824] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:49.875298 28776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94: No bootstrap required, opened a new log
I20260927 10:48:49.877696 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:49.864796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.10.96.254:32975)
0927 10:48:49.865241 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:49.865259 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:49.865308 (+ 49us) server_negotiation.cc:407] Connection header received
0927 10:48:49.866018 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:49.866035 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:49.866370 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:49.866727 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:49.867561 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:49.868566 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:49.869441 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:49.869962 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:49.870131 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:49.872856 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:49.872865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:49.872889 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:49.872919 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:49.874737 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:49.875183 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:49.875189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:49.875195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:49.875311 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:49.875667 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:49.875669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:49.875670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:49.876075 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:49.876207 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:49.876564 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 10:48:49.876903 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":150,"threads_started":1}
I20260927 10:48:49.879232 28776 raft_consensus.cc:348] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c577faefd4f61b509a3eecf11bb94" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32975 } }
I20260927 10:48:49.879472 28776 raft_consensus.cc:374] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:49.879539 28776 raft_consensus.cc:729] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45c577faefd4f61b509a3eecf11bb94, State: Initialized, Role: FOLLOWER
I20260927 10:48:49.879987 28776 consensus_queue.cc:260] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c577faefd4f61b509a3eecf11bb94" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32975 } }
I20260927 10:48:49.880108 28776 raft_consensus.cc:388] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:49.880190 28776 raft_consensus.cc:482] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:49.880311 28776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:49.881176 28776 raft_consensus.cc:504] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c577faefd4f61b509a3eecf11bb94" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32975 } }
I20260927 10:48:49.881567 28776 leader_election.cc:302] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d45c577faefd4f61b509a3eecf11bb94; no voters:
I20260927 10:48:49.881867 28776 leader_election.cc:288] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:49.882087 28781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:49.882349 28781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [term 1 LEADER]: Becoming Leader. State: Replica: d45c577faefd4f61b509a3eecf11bb94, State: Running, Role: LEADER
I20260927 10:48:49.882659 28781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c577faefd4f61b509a3eecf11bb94" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32975 } }
I20260927 10:48:49.883102 28776 sys_catalog.cc:564] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:49.884262 28783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45c577faefd4f61b509a3eecf11bb94. Latest consensus state: current_term: 1 leader_uuid: "d45c577faefd4f61b509a3eecf11bb94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c577faefd4f61b509a3eecf11bb94" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32975 } } }
I20260927 10:48:49.884398 28783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:49.884718 28782 sys_catalog.cc:455] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d45c577faefd4f61b509a3eecf11bb94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d45c577faefd4f61b509a3eecf11bb94" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 32975 } } }
I20260927 10:48:49.884842 28782 sys_catalog.cc:458] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:49.884796 28790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:49.887633 28790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:49.893760 28790 catalog_manager.cc:1347] Generated new cluster ID: 644d28778dc04659add535592b23fb58
I20260927 10:48:49.893833 28790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:49.907217 28790 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:49.908524 28790 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:49.913375 28790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94: Generated new TSK 0
I20260927 10:48:49.914273 28790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:48:49.946738 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32975
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:50.059165 28802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:50.059473 28802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:50.059548 28802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:50.063612 28802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:50.063702 28802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:50.063807 28802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:50.069118 28802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:50.070218 28802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:50.071269 28802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:50.080564 28808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.080555 28810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.080556 28807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:50.080662 28802 server_base.cc:1034] running on GCE node
I20260927 10:48:50.081243 28802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:50.081939 28802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:50.083117 28802 hybrid_clock.cc:648] HybridClock initialized: now 1790506130083086 us; error 48 us; skew 500 ppm
Sep 27 10:48:50 dist-test-slave-wmg0 krb5kdc[28695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506130, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:50.086179 28802 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:48:50.086993 28802 webserver.cc:460] Webserver started at http://127.10.96.193:44149/ using document root <none> and password file <none>
I20260927 10:48:50.087543 28802 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:50.087600 28802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:50.087762 28802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:50.089813 28802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "52cbcb7f82b74fa4914f0b152d5fa859"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "6d22bbd06fe3389651f8be14334db911"
server_key_iv: "e1d7bd8f197b37a791cfd1eea3b93cc0"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.090315 28802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "52cbcb7f82b74fa4914f0b152d5fa859"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "6d22bbd06fe3389651f8be14334db911"
server_key_iv: "e1d7bd8f197b37a791cfd1eea3b93cc0"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.093938 28802 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:48:50.096220 28817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.097397 28802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:50.097579 28802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "52cbcb7f82b74fa4914f0b152d5fa859"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "6d22bbd06fe3389651f8be14334db911"
server_key_iv: "e1d7bd8f197b37a791cfd1eea3b93cc0"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.097667 28802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:50.136770 28802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:50.140288 28802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:50.140450 28802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:50.141006 28802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:50.141891 28802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:50.141937 28802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.141979 28802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:50.142004 28802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.150807 28802 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:34179
I20260927 10:48:50.150827 28930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:34179 every 8 connection(s)
I20260927 10:48:50.151741 28802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:50.153999 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28802
I20260927 10:48:50.154163 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:48:50.154479 10627 external_mini_cluster.cc:1428] Setting key 470891fa45c912bc7bd2943e1967933b
Sep 27 10:48:50 dist-test-slave-wmg0 krb5kdc[28695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506130, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:48:50.167455 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.153560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:47609 (local address 127.10.96.254:32975)
0927 10:48:50.153726 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.153731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.154619 (+ 888us) server_negotiation.cc:407] Connection header received
0927 10:48:50.156200 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.156205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.156261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.156362 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:50.157641 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.158509 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.159725 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.159969 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.160084 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.162962 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:50.162965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:50.162967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:50.162991 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:50.164639 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.165196 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.165199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.165201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.165284 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.165801 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.165804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.165805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.165996 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:50.166113 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.166998 (+ 885us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.167192 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:48:50.168033 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.153960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32975 (local address 127.10.96.193:47609)
0927 10:48:50.154461 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.154516 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.155993 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.156555 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.156562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.156925 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:50.157423 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.157438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.158643 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.158645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.159569 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.159578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.160119 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.160123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.160310 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.160989 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:50.161005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:50.162778 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:50.164776 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:50.164781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:50.164794 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:50.165067 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:50.165403 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:50.165405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:50.165407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:50.165666 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:50.166106 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:50.166112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:50.166471 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.166740 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.167065 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":165,"threads_started":1}
I20260927 10:48:50.169554 28931 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32975
I20260927 10:48:50.169821 28931 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:50.170436 28931 heartbeater.cc:511] Master 127.10.96.254:32975 requested a full tablet report, sending...
I20260927 10:48:50.172158 28740 ts_manager.cc:194] Registered new tserver with Master: 52cbcb7f82b74fa4914f0b152d5fa859 (127.10.96.193:34179)
I20260927 10:48:50.173345 28740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:47609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:48:50.183827 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32975
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:50.296378 28936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:50.296669 28936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:50.296712 28936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:50.301059 28936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:50.301132 28936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:50.301195 28936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:50.307067 28936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:50.308161 28936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:50.309139 28936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:50.319101 28941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.319222 28942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:50.319309 28936 server_base.cc:1034] running on GCE node
W20260927 10:48:50.319197 28944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:50.319846 28936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:50.320504 28936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:50.321715 28936 hybrid_clock.cc:648] HybridClock initialized: now 1790506130321682 us; error 52 us; skew 500 ppm
Sep 27 10:48:50 dist-test-slave-wmg0 krb5kdc[28695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506130, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:50.324867 28936 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:48:50.326017 28936 webserver.cc:460] Webserver started at http://127.10.96.194:38381/ using document root <none> and password file <none>
I20260927 10:48:50.326619 28936 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:50.326721 28936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:50.326956 28936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:50.329064 28936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "cfd95c840eca4b16b4913c15273377cb"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "f5ef3e8eb4ac04fb779de3d5e88cf082"
server_key_iv: "d603c4464d01ccdad65c1522a32f601e"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.329690 28936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "cfd95c840eca4b16b4913c15273377cb"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "f5ef3e8eb4ac04fb779de3d5e88cf082"
server_key_iv: "d603c4464d01ccdad65c1522a32f601e"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.333657 28936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 10:48:50.336126 28951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.337582 28936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:50.337724 28936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "cfd95c840eca4b16b4913c15273377cb"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "f5ef3e8eb4ac04fb779de3d5e88cf082"
server_key_iv: "d603c4464d01ccdad65c1522a32f601e"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.337842 28936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:50.365236 28936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:50.368963 28936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:50.369184 28936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:50.369870 28936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:50.370736 28936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:50.370805 28936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.370872 28936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:50.370919 28936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.381749 28936 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:44451
I20260927 10:48:50.381803 29064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:44451 every 8 connection(s)
I20260927 10:48:50.383061 28936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:50.391013 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 28936
I20260927 10:48:50.391185 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:48:50.391556 10627 external_mini_cluster.cc:1428] Setting key dfc514a49e862ed15db7c9ffc2a6daa8
Sep 27 10:48:50 dist-test-slave-wmg0 krb5kdc[28695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506130, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:50.397938 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.385348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:56137 (local address 127.10.96.254:32975)
0927 10:48:50.385510 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.385515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.386843 (+ 1328us) server_negotiation.cc:407] Connection header received
0927 10:48:50.387728 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.387731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.387780 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.387865 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:50.389004 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.389641 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.390874 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.391136 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.391276 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.394181 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:50.394185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:50.394190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:50.394220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:50.395724 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.396278 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.396282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.396284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.396366 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.396734 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.396739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.396741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.396924 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:50.397043 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.397615 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.397766 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 10:48:50.398846 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.385798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32975 (local address 127.10.96.194:56137)
0927 10:48:50.386691 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.386733 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.387514 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.388021 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.388028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.388377 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:50.388850 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.388863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.389784 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.389787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.390746 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.390758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.391355 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.391359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.391586 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.392399 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:50.392415 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:50.394015 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:50.395857 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:50.395864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:50.395880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:50.396168 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:50.396469 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:50.396472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:50.396474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:50.396633 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:50.397033 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:50.397040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:50.397294 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.397572 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.397908 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":552,"threads_started":1}
I20260927 10:48:50.400117 29065 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32975
I20260927 10:48:50.400393 29065 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:50.401062 29065 heartbeater.cc:511] Master 127.10.96.254:32975 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
I20260927 10:48:50.402360 28740 ts_manager.cc:194] Registered new tserver with Master: cfd95c840eca4b16b4913c15273377cb (127.10.96.194:44451)
I20260927 10:48:50.402980 28740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:56137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:48:50.420625 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32975
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:50.529717 29070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:50.530026 29070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:50.530104 29070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:50.534137 29070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:50.534227 29070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:50.534322 29070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:50.539568 29070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:32975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:50.540738 29070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:50.541852 29070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:50.551628 29076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.551697 29075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.551563 29078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:50.551597 29070 server_base.cc:1034] running on GCE node
I20260927 10:48:50.552186 29070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:50.552819 29070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:50.553974 29070 hybrid_clock.cc:648] HybridClock initialized: now 1790506130553956 us; error 30 us; skew 500 ppm
Sep 27 10:48:50 dist-test-slave-wmg0 krb5kdc[28695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506130, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:50.556793 29070 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:48:50.557793 29070 webserver.cc:460] Webserver started at http://127.10.96.195:37731/ using document root <none> and password file <none>
I20260927 10:48:50.558357 29070 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:50.558415 29070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:50.558670 29070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:50.560708 29070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "cb801fb5771349e293980f8c395ac5b8"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "13b0462db22fdac0ab3299d572351511"
server_key_iv: "54722386fa19c83b15e8be20be647a84"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.561269 29070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "cb801fb5771349e293980f8c395ac5b8"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "13b0462db22fdac0ab3299d572351511"
server_key_iv: "54722386fa19c83b15e8be20be647a84"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.564896 29070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:50.567379 29085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.568607 29070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:50.568725 29070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "cb801fb5771349e293980f8c395ac5b8"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "13b0462db22fdac0ab3299d572351511"
server_key_iv: "54722386fa19c83b15e8be20be647a84"
server_key_version: "encryptionkey@0"
I20260927 10:48:50.568827 29070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:50.585765 29070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:50.588974 29070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:50.589193 29070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:50.589831 29070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:50.590679 29070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:50.590726 29070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.590767 29070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:50.590827 29070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:50.599970 29070 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:44243
I20260927 10:48:50.600015 29198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:44243 every 8 connection(s)
I20260927 10:48:50.600909 29070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:50.607016 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29070
I20260927 10:48:50.607164 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:48:50.607479 10627 external_mini_cluster.cc:1428] Setting key 399a6c079805f0ea8118b3ff581f3f3b
Sep 27 10:48:50 dist-test-slave-wmg0 krb5kdc[28695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506130, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:50.615067 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.602825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:45911 (local address 127.10.96.254:32975)
0927 10:48:50.602980 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.602984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.603839 (+ 855us) server_negotiation.cc:407] Connection header received
0927 10:48:50.605161 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.605165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.605222 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.605332 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:50.606516 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.607052 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.608645 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.608972 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.609070 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.611566 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:50.611569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:50.611571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:50.611594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:50.612912 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.613370 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.613372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.613373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.613441 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.613874 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.613876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.613877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.614049 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:50.614131 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.614835 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.614943 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:48:50.616043 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.603205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:32975 (local address 127.10.96.195:45911)
0927 10:48:50.603686 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.603728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.604950 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.605563 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.605570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.605912 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:50.606367 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.606380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.607216 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.607222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.608517 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.608528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.609120 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.609123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.609268 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.609884 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:50.609901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:50.611398 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:50.613020 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:50.613025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:50.613037 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:50.613285 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:50.613579 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:50.613583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:50.613585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:50.613784 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:50.614181 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:50.614186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:50.614450 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.614775 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.615053 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":106,"threads_started":1}
I20260927 10:48:50.617200 29199 heartbeater.cc:347] Connected to a master server at 127.10.96.254:32975
I20260927 10:48:50.617532 29199 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:50.618047 29199 heartbeater.cc:511] Master 127.10.96.254:32975 requested a full tablet report, sending...
I20260927 10:48:50.619148 28740 ts_manager.cc:194] Registered new tserver with Master: cb801fb5771349e293980f8c395ac5b8 (127.10.96.195:44243)
I20260927 10:48:50.619765 28740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:45911
I20260927 10:48:50.621641 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:50.631784 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.623809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.10.96.254:32975)
0927 10:48:50.623933 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.623937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.624036 (+ 99us) server_negotiation.cc:407] Connection header received
0927 10:48:50.624128 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.624130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.624168 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.624240 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:50.624987 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.625611 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.626476 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.626695 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.626778 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.628341 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:50.628344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:50.628346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:50.628370 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:50.629972 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.630614 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.630617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.630619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.630697 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:50.631025 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:50.631027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:50.631028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:50.631262 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:50.631378 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.631581 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.631660 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:48:50.634899 28740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:50.638003 28740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:50.652874 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.647095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:44451 (local address 127.0.0.1:58132)
0927 10:48:50.647780 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.647793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.647865 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.648575 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.648577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.648591 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:50.648662 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.648666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.649944 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.649949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.651271 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.651282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.652319 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.652322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.652425 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.652466 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.652584 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.652720 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":102,"threads_started":1}
I20260927 10:48:50.653026 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.647401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.10.96.194:44451)
0927 10:48:50.648171 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.648184 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.648206 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:48:50.648277 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.648280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.648444 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.648589 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:50.648738 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.649805 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.651396 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.652193 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.652310 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.652460 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.652618 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.652834 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":109,"threads_started":1}
I20260927 10:48:50.653193 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.646355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:44243 (local address 127.0.0.1:54928)
0927 10:48:50.646998 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.647048 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.647183 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.648029 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.648036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.648058 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:50.648287 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.648294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.650918 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.650922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.651804 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.651811 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.652770 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.652774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.652873 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.652898 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.652972 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.653042 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":50,"spinlock_wait_cycles":9984,"thread_start_us":99,"threads_started":1}
I20260927 10:48:50.653661 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.646498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.10.96.195:44243)
0927 10:48:50.647125 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.647140 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.647168 (+ 28us) server_negotiation.cc:407] Connection header received
0927 10:48:50.647429 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.647436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.647659 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.647818 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:50.649590 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.650776 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.651926 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.652648 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.652747 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.652874 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.653402 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.653533 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":115,"threads_started":1}
I20260927 10:48:50.653892 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.646429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34179 (local address 127.0.0.1:39398)
0927 10:48:50.647399 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.647413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.647499 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.648047 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.648049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.648071 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:50.648287 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.648294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.649793 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.649797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.651179 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.651188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.653197 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.653201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.653581 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.653605 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.653677 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.653742 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":102,"spinlock_wait_cycles":2432,"thread_start_us":104,"threads_started":1}
I20260927 10:48:50.654693 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.646800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.10.96.193:34179)
0927 10:48:50.647138 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.647152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.647627 (+ 475us) server_negotiation.cc:407] Connection header received
0927 10:48:50.647710 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.647716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.647912 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.648068 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:50.648407 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.649655 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.651314 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.652193 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.652310 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.652462 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.654459 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.654561 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":78,"threads_started":1}
I20260927 10:48:50.655972 28999 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2a6e90553a4690a8146d8414932de9 (DEFAULT_TABLE table=test-table [id=9378b638735f4a54b5c6d40654be8947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:50.656304 29133 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2a6e90553a4690a8146d8414932de9 (DEFAULT_TABLE table=test-table [id=9378b638735f4a54b5c6d40654be8947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:50.656857 28999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2a6e90553a4690a8146d8414932de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:50.657280 29133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2a6e90553a4690a8146d8414932de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:50.657233 28865 tablet_service.cc:1467] Processing CreateTablet for tablet 7e2a6e90553a4690a8146d8414932de9 (DEFAULT_TABLE table=test-table [id=9378b638735f4a54b5c6d40654be8947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:50.658176 28865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e2a6e90553a4690a8146d8414932de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:50.663120 29216 tablet_bootstrap.cc:492] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Bootstrap starting.
I20260927 10:48:50.664713 29217 tablet_bootstrap.cc:492] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Bootstrap starting.
I20260927 10:48:50.665081 29218 tablet_bootstrap.cc:492] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859: Bootstrap starting.
I20260927 10:48:50.665767 29216 tablet_bootstrap.cc:654] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:50.666746 29217 tablet_bootstrap.cc:654] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:50.667277 29216 log.cc:824] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:50.667399 29218 tablet_bootstrap.cc:654] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:50.667809 29217 log.cc:824] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:50.668609 29218 log.cc:824] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:50.669374 29216 tablet_bootstrap.cc:492] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: No bootstrap required, opened a new log
I20260927 10:48:50.669587 29216 ts_tablet_manager.cc:1397] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:48:50.669904 29217 tablet_bootstrap.cc:492] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: No bootstrap required, opened a new log
I20260927 10:48:50.670110 29217 ts_tablet_manager.cc:1397] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 10:48:50.670447 29218 tablet_bootstrap.cc:492] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859: No bootstrap required, opened a new log
I20260927 10:48:50.670639 29218 ts_tablet_manager.cc:1397] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:48:50.673417 29217 raft_consensus.cc:348] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.673429 29216 raft_consensus.cc:348] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.673692 29216 raft_consensus.cc:374] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:50.673708 29217 raft_consensus.cc:374] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:50.673758 29216 raft_consensus.cc:729] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd95c840eca4b16b4913c15273377cb, State: Initialized, Role: FOLLOWER
I20260927 10:48:50.673764 29217 raft_consensus.cc:729] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb801fb5771349e293980f8c395ac5b8, State: Initialized, Role: FOLLOWER
I20260927 10:48:50.674199 29217 consensus_queue.cc:260] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.674203 29216 consensus_queue.cc:260] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.674571 29218 raft_consensus.cc:348] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.674811 29199 heartbeater.cc:503] Master 127.10.96.254:32975 was elected leader, sending a full tablet report...
I20260927 10:48:50.674825 29218 raft_consensus.cc:374] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:50.674876 29218 raft_consensus.cc:729] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52cbcb7f82b74fa4914f0b152d5fa859, State: Initialized, Role: FOLLOWER
I20260927 10:48:50.675181 29216 ts_tablet_manager.cc:1428] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:50.675196 29065 heartbeater.cc:503] Master 127.10.96.254:32975 was elected leader, sending a full tablet report...
I20260927 10:48:50.675423 29218 consensus_queue.cc:260] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.675989 28931 heartbeater.cc:503] Master 127.10.96.254:32975 was elected leader, sending a full tablet report...
I20260927 10:48:50.676621 29217 ts_tablet_manager.cc:1428] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:50.677392 29218 ts_tablet_manager.cc:1428] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 10:48:50.751725 29223 raft_consensus.cc:482] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:50.752243 29223 raft_consensus.cc:504] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.753423 29223 leader_election.cc:288] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb801fb5771349e293980f8c395ac5b8 (127.10.96.195:44243), 52cbcb7f82b74fa4914f0b152d5fa859 (127.10.96.193:34179)
I20260927 10:48:50.757148 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.753730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34179 (local address 127.10.96.194:56877)
0927 10:48:50.753833 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.753850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.753947 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.754320 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.754323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.754342 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:50.754464 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.754470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.755235 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.755240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.756047 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.756055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.756708 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.756711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.756821 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.756848 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.756938 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.757012 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260927 10:48:50.757148 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.753957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:56877 (local address 127.10.96.193:34179)
0927 10:48:50.754090 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.754094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.754107 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:48:50.754156 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.754159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.754199 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.754305 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:50.754595 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.755116 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.756179 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.756584 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.756713 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.756766 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.756955 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.757032 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 10:48:50.757571 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.753829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:48589 (local address 127.10.96.195:44243)
0927 10:48:50.753952 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.753955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.754458 (+ 503us) server_negotiation.cc:407] Connection header received
0927 10:48:50.754595 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.754598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.754649 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.754728 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:50.755040 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.755525 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.756559 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.756964 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.757095 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.757143 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.757344 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.757420 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:48:50.757916 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.753730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:44243 (local address 127.10.96.194:48589)
0927 10:48:50.754355 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.754366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:50.754483 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:50.754749 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:50.754752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:50.754769 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:50.754896 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.754900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.755634 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.755639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.756436 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:50.756444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.757083 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.757087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:50.757181 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.757204 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 10:48:50.757741 (+ 537us) client_negotiation.cc:240] Negotiation successful
0927 10:48:50.757814 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":288,"thread_start_us":134,"threads_started":1}
I20260927 10:48:50.757905 28885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2a6e90553a4690a8146d8414932de9" candidate_uuid: "cfd95c840eca4b16b4913c15273377cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" is_pre_election: true
I20260927 10:48:50.758231 28885 raft_consensus.cc:2454] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd95c840eca4b16b4913c15273377cb in term 0.
I20260927 10:48:50.758621 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2a6e90553a4690a8146d8414932de9" candidate_uuid: "cfd95c840eca4b16b4913c15273377cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb801fb5771349e293980f8c395ac5b8" is_pre_election: true
I20260927 10:48:50.758728 28952 leader_election.cc:302] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52cbcb7f82b74fa4914f0b152d5fa859, cfd95c840eca4b16b4913c15273377cb; no voters:
I20260927 10:48:50.758946 29153 raft_consensus.cc:2454] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd95c840eca4b16b4913c15273377cb in term 0.
I20260927 10:48:50.759043 29223 raft_consensus.cc:2781] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:50.759133 29223 raft_consensus.cc:482] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:50.759207 29223 raft_consensus.cc:3037] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:50.760260 29223 raft_consensus.cc:504] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.760639 29223 leader_election.cc:288] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [CANDIDATE]: Term 1 election: Requested vote from peers cb801fb5771349e293980f8c395ac5b8 (127.10.96.195:44243), 52cbcb7f82b74fa4914f0b152d5fa859 (127.10.96.193:34179)
I20260927 10:48:50.760984 29153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2a6e90553a4690a8146d8414932de9" candidate_uuid: "cfd95c840eca4b16b4913c15273377cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb801fb5771349e293980f8c395ac5b8"
I20260927 10:48:50.761020 28885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e2a6e90553a4690a8146d8414932de9" candidate_uuid: "cfd95c840eca4b16b4913c15273377cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52cbcb7f82b74fa4914f0b152d5fa859"
I20260927 10:48:50.761093 29153 raft_consensus.cc:3037] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:50.761178 28885 raft_consensus.cc:3037] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:50.762231 29153 raft_consensus.cc:2454] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd95c840eca4b16b4913c15273377cb in term 1.
I20260927 10:48:50.762425 28885 raft_consensus.cc:2454] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd95c840eca4b16b4913c15273377cb in term 1.
I20260927 10:48:50.762578 28953 leader_election.cc:302] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb801fb5771349e293980f8c395ac5b8, cfd95c840eca4b16b4913c15273377cb; no voters:
I20260927 10:48:50.762786 29223 raft_consensus.cc:2781] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:50.763051 29223 raft_consensus.cc:686] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 1 LEADER]: Becoming Leader. State: Replica: cfd95c840eca4b16b4913c15273377cb, State: Running, Role: LEADER
I20260927 10:48:50.763389 29223 consensus_queue.cc:237] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } }
I20260927 10:48:50.766536 28737 catalog_manager.cc:5495] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb reported cstate change: term changed from 0 to 1, leader changed from <none> to cfd95c840eca4b16b4913c15273377cb (127.10.96.194). New cstate: current_term: 1 leader_uuid: "cfd95c840eca4b16b4913c15273377cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfd95c840eca4b16b4913c15273377cb" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 44451 } health_report { overall_health: HEALTHY } } }
I20260927 10:48:50.779211 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.775636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.10.96.194:44451)
0927 10:48:50.775773 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:50.775777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:50.775790 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:48:50.775851 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:50.775853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:50.775921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:50.776013 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:50.776234 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.776740 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.777721 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:50.777944 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:50.778035 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:50.778398 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:50.778807 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:50.778931 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:50.779050 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 10:48:50.779086 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 10:48:50.785295 28885 raft_consensus.cc:1261] T 7e2a6e90553a4690a8146d8414932de9 P 52cbcb7f82b74fa4914f0b152d5fa859 [term 1 FOLLOWER]: Refusing update from remote peer cfd95c840eca4b16b4913c15273377cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:50.785295 29153 raft_consensus.cc:1261] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 1 FOLLOWER]: Refusing update from remote peer cfd95c840eca4b16b4913c15273377cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:50.786083 29226 consensus_queue.cc:1035] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "52cbcb7f82b74fa4914f0b152d5fa859" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:50.786270 29223 consensus_queue.cc:1035] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb801fb5771349e293980f8c395ac5b8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 44243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:50.792483 29229 mvcc.cc:205] Tried to move back new op lower bound from 7333913111688892416 to 7333913111609360384. Current Snapshot: MvccSnapshot[applied={T|T < 7333913111688892416}]
I20260927 10:48:50.803349 28740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:50.803573 28740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:50.806169 28740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d45c577faefd4f61b509a3eecf11bb94: Sending DeleteTablet for 3 replicas of tablet 7e2a6e90553a4690a8146d8414932de9
I20260927 10:48:50.807011 28865 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e2a6e90553a4690a8146d8414932de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:50 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:39398
I20260927 10:48:50.807011 29133 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e2a6e90553a4690a8146d8414932de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:50 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:54928
I20260927 10:48:50.807123 28999 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e2a6e90553a4690a8146d8414932de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:50 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:58132
I20260927 10:48:50.807595 29239 tablet_replica.cc:330] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: stopping tablet replica
I20260927 10:48:50.807945 29239 raft_consensus.cc:2229] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:50.808211 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28802
I20260927 10:48:50.808295 29239 raft_consensus.cc:2258] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:50.810403 29241 tablet_replica.cc:330] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: stopping tablet replica
I20260927 10:48:50.810437 29239 ts_tablet_manager.cc:1905] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:50.810782 29241 raft_consensus.cc:2229] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:48:50.811269 29241 raft_consensus.cc:2258] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:50.813194 29241 ts_tablet_manager.cc:1905] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:50.813690 29239 ts_tablet_manager.cc:1918] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:50.813834 29239 log.cc:1196] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/7e2a6e90553a4690a8146d8414932de9
I20260927 10:48:50.814205 29239 ts_tablet_manager.cc:1939] T 7e2a6e90553a4690a8146d8414932de9 P cb801fb5771349e293980f8c395ac5b8: Deleting consensus metadata
I20260927 10:48:50.815603 28725 catalog_manager.cc:4841] TS cb801fb5771349e293980f8c395ac5b8 (127.10.96.195:44243): tablet 7e2a6e90553a4690a8146d8414932de9 (table test-table [id=9378b638735f4a54b5c6d40654be8947]) successfully deleted
I20260927 10:48:50.816336 29241 ts_tablet_manager.cc:1918] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:50.816463 29241 log.cc:1196] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/7e2a6e90553a4690a8146d8414932de9
I20260927 10:48:50.816761 29241 ts_tablet_manager.cc:1939] T 7e2a6e90553a4690a8146d8414932de9 P cfd95c840eca4b16b4913c15273377cb: Deleting consensus metadata
W20260927 10:48:50.817876 28724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:34179 (error 108)
I20260927 10:48:50.818029 28725 catalog_manager.cc:4841] TS cfd95c840eca4b16b4913c15273377cb (127.10.96.194:44451): tablet 7e2a6e90553a4690a8146d8414932de9 (table test-table [id=9378b638735f4a54b5c6d40654be8947]) successfully deleted
I20260927 10:48:50.818701 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28936
I20260927 10:48:50.818934 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:50.818619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:34179 (local address 127.0.0.1:39412)
0927 10:48:50.818752 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:50.818827 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:34179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 10:48:50.819550 28724 catalog_manager.cc:4584] TS 52cbcb7f82b74fa4914f0b152d5fa859 (127.10.96.193:34179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e2a6e90553a4690a8146d8414932de9: Network error: Client connection negotiation failed: client connection to 127.10.96.193:34179: connect: Connection refused (error 111)
I20260927 10:48:50.824728 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29070
I20260927 10:48:50.830878 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 28708
2026-09-27T10:48:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2322 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T10:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:50Z Disabled control of system clock
I20260927 10:48:50.861917 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:39793
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:48:50.974673 29246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:50.974941 29246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:50.974979 29246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:50.979049 29246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:50.979111 29246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:50.979127 29246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:50.979144 29246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:50.979159 29246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:50.983880 29246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:39793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:50.984948 29246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:50.986315 29246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:50.994741 29251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.994740 29254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:50.994764 29252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:50.994974 29246 server_base.cc:1034] running on GCE node
I20260927 10:48:50.995664 29246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:50.996730 29246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:50.997927 29246 hybrid_clock.cc:648] HybridClock initialized: now 1790506130997894 us; error 47 us; skew 500 ppm
I20260927 10:48:50.999766 29246 webserver.cc:460] Webserver started at http://127.10.96.254:40795/ using document root <none> and password file <none>
I20260927 10:48:51.000350 29246 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:51.000439 29246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:51.000679 29246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:51.002766 29246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "28660fb6583e4f5ba273328de021225d"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "f94df93d583ebf3a6b62d459cd89c315"
server_key_iv: "d466a9821deb559491d07121c6dc5a9e"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.003429 29246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "28660fb6583e4f5ba273328de021225d"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "f94df93d583ebf3a6b62d459cd89c315"
server_key_iv: "d466a9821deb559491d07121c6dc5a9e"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.007303 29246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 10:48:51.009693 29260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.010951 29246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:51.011090 29246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "28660fb6583e4f5ba273328de021225d"
format_stamp: "Formatted at 2026-09-27 10:48:50 on dist-test-slave-wmg0"
server_key: "f94df93d583ebf3a6b62d459cd89c315"
server_key_iv: "d466a9821deb559491d07121c6dc5a9e"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.011204 29246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:51.030959 29246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:51.031734 29246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:51.031944 29246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:51.039225 29246 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:39793
I20260927 10:48:51.039232 29312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:39793 every 8 connection(s)
I20260927 10:48:51.040196 29246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:51.043447 29313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:51.047963 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29246
I20260927 10:48:51.048132 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:51.048417 10627 external_mini_cluster.cc:1428] Setting key d367d317721495104148fe73e7a3e93f
I20260927 10:48:51.048923 29313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d: Bootstrap starting.
I20260927 10:48:51.051316 29313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:51.052160 29313 log.cc:824] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:51.054517 29313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d: No bootstrap required, opened a new log
I20260927 10:48:51.058207 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.049910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.10.96.254:39793)
0927 10:48:51.050523 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.050537 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.050580 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:48:51.051200 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.051217 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.051661 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.051935 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:51.052339 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.053582 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.054768 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.055245 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.055367 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.056304 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:51.056309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:51.056323 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:51.056741 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:51.056797 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:51.056816 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:51.056895 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.057069 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.057382 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":134,"threads_started":1}
I20260927 10:48:51.058830 29313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28660fb6583e4f5ba273328de021225d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39793 } }
I20260927 10:48:51.059052 29313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:51.059111 29313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28660fb6583e4f5ba273328de021225d, State: Initialized, Role: FOLLOWER
I20260927 10:48:51.059578 29313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28660fb6583e4f5ba273328de021225d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39793 } }
I20260927 10:48:51.059734 29313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:51.059819 29313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:51.059943 29313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:51.061049 29313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28660fb6583e4f5ba273328de021225d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39793 } }
I20260927 10:48:51.061384 29313 leader_election.cc:302] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28660fb6583e4f5ba273328de021225d; no voters:
I20260927 10:48:51.061714 29313 leader_election.cc:288] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:51.061848 29318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:51.062064 29318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [term 1 LEADER]: Becoming Leader. State: Replica: 28660fb6583e4f5ba273328de021225d, State: Running, Role: LEADER
I20260927 10:48:51.062378 29318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28660fb6583e4f5ba273328de021225d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39793 } }
I20260927 10:48:51.063014 29313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:51.063974 29320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28660fb6583e4f5ba273328de021225d. Latest consensus state: current_term: 1 leader_uuid: "28660fb6583e4f5ba273328de021225d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28660fb6583e4f5ba273328de021225d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39793 } } }
I20260927 10:48:51.064093 29320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:51.064458 29327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:51.064862 29319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28660fb6583e4f5ba273328de021225d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28660fb6583e4f5ba273328de021225d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 39793 } } }
I20260927 10:48:51.064972 29319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:51.068216 29327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:51.074263 29327 catalog_manager.cc:1347] Generated new cluster ID: 2f5060c9cc9742e0a1b2349406f162fd
I20260927 10:48:51.074340 29327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:51.085224 29327 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:51.086228 29327 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:51.094210 29327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d: Generated new TSK 0
I20260927 10:48:51.094928 29327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:51.100986 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39793
--builtin_ntp_servers=127.10.96.212:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:48:51.212951 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:51.213219 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:51.213258 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:51.217615 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:51.217670 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:51.217727 29337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:51.222632 29337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:51.223775 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:51.225204 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:51.234830 29342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:51.234831 29343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:51.234830 29345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:51.234989 29337 server_base.cc:1034] running on GCE node
I20260927 10:48:51.235523 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:51.236155 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:51.237362 29337 hybrid_clock.cc:648] HybridClock initialized: now 1790506131237341 us; error 41 us; skew 500 ppm
I20260927 10:48:51.239320 29337 webserver.cc:460] Webserver started at http://127.10.96.193:43033/ using document root <none> and password file <none>
I20260927 10:48:51.239909 29337 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:51.240010 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:51.240238 29337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:51.242354 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "07e73a50fd61462f84ce3878f864fd5e"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "b3c9ec3b2065102f9c421243ec379003"
server_key_iv: "bfacc40c64881934e6ffa18558f823c8"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.242913 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "07e73a50fd61462f84ce3878f864fd5e"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "b3c9ec3b2065102f9c421243ec379003"
server_key_iv: "bfacc40c64881934e6ffa18558f823c8"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.246598 29337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:51.249158 29351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.250432 29337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:51.250582 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "07e73a50fd61462f84ce3878f864fd5e"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "b3c9ec3b2065102f9c421243ec379003"
server_key_iv: "bfacc40c64881934e6ffa18558f823c8"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.250705 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:51.270270 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:51.271167 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:51.271323 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:51.271852 29337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:51.272743 29337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:51.272789 29337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.272830 29337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:51.272854 29337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.281762 29337 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39427
I20260927 10:48:51.281778 29464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39427 every 8 connection(s)
I20260927 10:48:51.282806 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:51.287770 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29337
I20260927 10:48:51.287927 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:48:51.288233 10627 external_mini_cluster.cc:1428] Setting key 99e3c6110a4f3a05b6683869c61dba29
I20260927 10:48:51.290321 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39793
--builtin_ntp_servers=127.10.96.212:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:48:51.293062 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.284887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:55471 (local address 127.10.96.254:39793)
0927 10:48:51.285072 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.285075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.286246 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 10:48:51.287686 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.287691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.287759 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.287881 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:51.289015 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.289650 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.290943 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.291198 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.291315 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.292543 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:51.292546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:51.292548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:51.292561 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:51.292580 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:51.292588 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:51.292658 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.292848 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.292908 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 10:48:51.294409 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.285312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39793 (local address 127.10.96.193:55471)
0927 10:48:51.286092 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.286136 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.287434 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.288177 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.288188 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.288309 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:51.288851 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.288866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.289780 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.289784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.290816 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.290826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.291359 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.291363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.291546 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.292232 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:51.292234 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:51.292277 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:51.292346 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:51.292681 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:51.292694 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:51.292703 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.292844 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.293192 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":179,"threads_started":1}
I20260927 10:48:51.295850 29465 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39793
I20260927 10:48:51.296150 29465 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:51.296721 29465 heartbeater.cc:511] Master 127.10.96.254:39793 requested a full tablet report, sending...
I20260927 10:48:51.298346 29277 ts_manager.cc:194] Registered new tserver with Master: 07e73a50fd61462f84ce3878f864fd5e (127.10.96.193:39427)
I20260927 10:48:51.299525 29277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.193:55471
W20260927 10:48:51.400578 29468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:51.400867 29468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:51.400941 29468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:51.405149 29468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:51.405233 29468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:51.405325 29468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:51.410068 29468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:51.411160 29468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:51.412580 29468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:51.422158 29473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:51.422170 29474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:51.422222 29468 server_base.cc:1034] running on GCE node
W20260927 10:48:51.422170 29476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:51.422770 29468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:51.423327 29468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:51.424499 29468 hybrid_clock.cc:648] HybridClock initialized: now 1790506131424466 us; error 46 us; skew 500 ppm
I20260927 10:48:51.426333 29468 webserver.cc:460] Webserver started at http://127.10.96.194:42141/ using document root <none> and password file <none>
I20260927 10:48:51.426910 29468 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:51.426975 29468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:51.427215 29468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:51.429229 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "89d66da5ffef4682820b98daffeaba7c"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "466cc8a99128cd686dd296a5c47ff3ab"
server_key_iv: "ab9619b201078ea07c06dba6919405bc"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.429852 29468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "89d66da5ffef4682820b98daffeaba7c"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "466cc8a99128cd686dd296a5c47ff3ab"
server_key_iv: "ab9619b201078ea07c06dba6919405bc"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.433437 29468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:51.435921 29482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.437136 29468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 10:48:51.437269 29468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "89d66da5ffef4682820b98daffeaba7c"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "466cc8a99128cd686dd296a5c47ff3ab"
server_key_iv: "ab9619b201078ea07c06dba6919405bc"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.437384 29468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:51.454233 29468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:51.454962 29468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:51.455171 29468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:51.455758 29468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:51.456736 29468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:51.456807 29468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.456872 29468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:51.456933 29468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.466481 29468 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:36581
I20260927 10:48:51.466559 29595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:36581 every 8 connection(s)
I20260927 10:48:51.467506 29468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:51.477175 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29468
I20260927 10:48:51.477322 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:48:51.477408 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.469520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:35781 (local address 127.10.96.254:39793)
0927 10:48:51.469689 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.469694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.470807 (+ 1113us) server_negotiation.cc:407] Connection header received
0927 10:48:51.472150 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.472153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.472197 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.472272 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:51.473468 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.473988 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.475405 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.475618 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.475709 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.476812 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:51.476815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:51.476817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:51.476829 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:51.476845 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:51.476853 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:51.476926 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.477155 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.477207 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 10:48:51.477706 10627 external_mini_cluster.cc:1428] Setting key 6c46e283bb02e74247f8bc8fee55d981
I20260927 10:48:51.478451 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.469977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39793 (local address 127.10.96.194:35781)
0927 10:48:51.470646 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.470687 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.471950 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.472442 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.472448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.472548 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:51.473303 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.473320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.474344 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.474347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.475273 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.475282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.475755 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.475757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.475912 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.476564 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:51.476566 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:51.476605 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:51.476669 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:51.476967 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:51.476986 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:51.476995 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.477152 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.477491 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260927 10:48:51.479792 29596 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39793
I20260927 10:48:51.480063 29596 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:51.480062 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39793
--builtin_ntp_servers=127.10.96.212:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:48:51.480607 29596 heartbeater.cc:511] Master 127.10.96.254:39793 requested a full tablet report, sending...
I20260927 10:48:51.481689 29277 ts_manager.cc:194] Registered new tserver with Master: 89d66da5ffef4682820b98daffeaba7c (127.10.96.194:36581)
I20260927 10:48:51.482246 29277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.194:35781
W20260927 10:48:51.592815 29599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:51.593123 29599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:51.593206 29599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:51.597445 29599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:51.597586 29599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:51.597678 29599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:51.602545 29599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:51.603652 29599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:51.605178 29599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:51.615317 29605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:51.615387 29604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:51.615429 29599 server_base.cc:1034] running on GCE node
W20260927 10:48:51.615317 29607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:51.616078 29599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:51.616683 29599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:51.617864 29599 hybrid_clock.cc:648] HybridClock initialized: now 1790506131617843 us; error 34 us; skew 500 ppm
I20260927 10:48:51.619954 29599 webserver.cc:460] Webserver started at http://127.10.96.195:36769/ using document root <none> and password file <none>
I20260927 10:48:51.620559 29599 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:51.620661 29599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:51.620900 29599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:51.623080 29599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "a90f24dcadb24a20872a50b9b9994a13"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "6dbbe7fd31ed69ce9dd48127468c3b75"
server_key_iv: "8faff505a39c29ee99d26e820f493201"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.623692 29599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "a90f24dcadb24a20872a50b9b9994a13"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "6dbbe7fd31ed69ce9dd48127468c3b75"
server_key_iv: "8faff505a39c29ee99d26e820f493201"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.627465 29599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:48:51.630113 29613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.631577 29599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:48:51.631716 29599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "a90f24dcadb24a20872a50b9b9994a13"
format_stamp: "Formatted at 2026-09-27 10:48:51 on dist-test-slave-wmg0"
server_key: "6dbbe7fd31ed69ce9dd48127468c3b75"
server_key_iv: "8faff505a39c29ee99d26e820f493201"
server_key_version: "encryptionkey@0"
I20260927 10:48:51.631850 29599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:51.647184 29599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:51.647878 29599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:51.648085 29599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:51.648671 29599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:51.649676 29599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:51.649745 29599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.649813 29599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:51.649859 29599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:51.658484 29599 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:41961
I20260927 10:48:51.658543 29726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:41961 every 8 connection(s)
I20260927 10:48:51.659408 29599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:51.666489 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29599
I20260927 10:48:51.666594 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:48:51.666877 10627 external_mini_cluster.cc:1428] Setting key 4791cdd71bc743e4b7feab0d6ca6115f
I20260927 10:48:51.669003 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.661206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:56115 (local address 127.10.96.254:39793)
0927 10:48:51.661347 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.661350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.662436 (+ 1086us) server_negotiation.cc:407] Connection header received
0927 10:48:51.663861 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.663864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.663975 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.664052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:51.664956 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.665657 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.666858 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.667158 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.667272 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.668501 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:51.668504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:51.668506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:51.668516 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:51.668536 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:51.668545 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:51.668606 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.668792 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.668860 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:48:51.669983 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.661653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:39793 (local address 127.10.96.195:56115)
0927 10:48:51.662288 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.662329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.663622 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.664211 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.664218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.664302 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:51.664797 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.664811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.665762 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.665764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.666741 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.666750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.667312 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.667316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.667521 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.668221 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:51.668223 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:51.668281 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:51.668363 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:51.668627 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:51.668636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:51.668641 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.668773 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.669068 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260927 10:48:51.671231 29727 heartbeater.cc:347] Connected to a master server at 127.10.96.254:39793
I20260927 10:48:51.671522 29727 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:51.672066 29727 heartbeater.cc:511] Master 127.10.96.254:39793 requested a full tablet report, sending...
I20260927 10:48:51.673110 29277 ts_manager.cc:194] Registered new tserver with Master: a90f24dcadb24a20872a50b9b9994a13 (127.10.96.195:41961)
I20260927 10:48:51.673799 29277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.195:56115
I20260927 10:48:51.680824 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:51.686407 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.682825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.10.96.254:39793)
0927 10:48:51.682971 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.682975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.683104 (+ 129us) server_negotiation.cc:407] Connection header received
0927 10:48:51.683210 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.683213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.683250 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.683319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:51.683521 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.684039 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.684855 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.685083 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.685158 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.686006 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:51.686009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:51.686011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:51.686022 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:51.686039 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:51.686047 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:51.686153 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.686268 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.686302 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 10:48:51.689594 29277 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:51.692612 29277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:51.707518 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.701893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.10.96.195:41961)
0927 10:48:51.702562 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.702573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.702595 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:48:51.702676 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.702684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.702873 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.703043 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:51.703271 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.704087 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.706050 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.706783 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.706896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.707072 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.707216 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.707321 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":138,"threads_started":1}
I20260927 10:48:51.707516 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.701115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:41961 (local address 127.0.0.1:53992)
0927 10:48:51.702432 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.702446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.702522 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.703019 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.703022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.703046 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:51.703151 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.703161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.704197 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.704202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.705918 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.705928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.706914 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.706918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.707022 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.707060 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.707166 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.707327 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":83,"thread_start_us":107,"threads_started":1}
I20260927 10:48:51.707741 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.701971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.10.96.194:36581)
0927 10:48:51.702352 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.702366 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.702710 (+ 344us) server_negotiation.cc:407] Connection header received
0927 10:48:51.702963 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.702970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.703157 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.703288 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:51.703519 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.704561 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.706124 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.706763 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.706829 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.707047 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.707458 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.707570 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":151,"threads_started":1}
I20260927 10:48:51.708153 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.701369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:36581 (local address 127.0.0.1:59784)
0927 10:48:51.702624 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.702638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.702782 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.703308 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.703312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.703331 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:51.703405 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.703410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.704702 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.704705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.705974 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.705984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.707161 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.707164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.707271 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.707323 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.707921 (+ 598us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.708022 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"mutex_wait_us":31,"spinlock_wait_cycles":768,"thread_start_us":125,"threads_started":1}
I20260927 10:48:51.708185 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.701210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.10.96.193:39427)
0927 10:48:51.701672 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.701686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.702254 (+ 568us) server_negotiation.cc:407] Connection header received
0927 10:48:51.702340 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.702345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.702561 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.702720 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:51.703057 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.704265 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.705960 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.706783 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.706896 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.707072 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.707698 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.707800 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":120,"threads_started":1}
I20260927 10:48:51.707597 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.701093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39427 (local address 127.0.0.1:58766)
0927 10:48:51.701900 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.701948 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.702098 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.702717 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.702723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.702745 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:51.702924 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.702933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.704376 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.704379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.705820 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.705831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.707155 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.707158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.707268 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.707312 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.707381 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.707450 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":37,"spinlock_wait_cycles":1408,"thread_start_us":63,"threads_started":1}
I20260927 10:48:51.710840 29399 tablet_service.cc:1467] Processing CreateTablet for tablet 83301c220415436fb46872b40da50eee (DEFAULT_TABLE table=test-table [id=0cc1ffe99d4c4058a1aa1de895157528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:51.710920 29661 tablet_service.cc:1467] Processing CreateTablet for tablet 83301c220415436fb46872b40da50eee (DEFAULT_TABLE table=test-table [id=0cc1ffe99d4c4058a1aa1de895157528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:51.710920 29530 tablet_service.cc:1467] Processing CreateTablet for tablet 83301c220415436fb46872b40da50eee (DEFAULT_TABLE table=test-table [id=0cc1ffe99d4c4058a1aa1de895157528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:51.711848 29399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83301c220415436fb46872b40da50eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:51.711910 29661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83301c220415436fb46872b40da50eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:51.711910 29530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83301c220415436fb46872b40da50eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:51.718719 29744 tablet_bootstrap.cc:492] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: Bootstrap starting.
I20260927 10:48:51.719645 29746 tablet_bootstrap.cc:492] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Bootstrap starting.
I20260927 10:48:51.719750 29745 tablet_bootstrap.cc:492] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: Bootstrap starting.
I20260927 10:48:51.721050 29744 tablet_bootstrap.cc:654] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:51.721431 29745 tablet_bootstrap.cc:654] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:51.721662 29746 tablet_bootstrap.cc:654] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:51.721949 29744 log.cc:824] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:51.722201 29745 log.cc:824] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:51.722431 29746 log.cc:824] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:51.724083 29744 tablet_bootstrap.cc:492] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: No bootstrap required, opened a new log
I20260927 10:48:51.724138 29746 tablet_bootstrap.cc:492] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: No bootstrap required, opened a new log
I20260927 10:48:51.724138 29745 tablet_bootstrap.cc:492] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: No bootstrap required, opened a new log
I20260927 10:48:51.724299 29744 ts_tablet_manager.cc:1397] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:48:51.724308 29746 ts_tablet_manager.cc:1397] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:48:51.724308 29745 ts_tablet_manager.cc:1397] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 10:48:51.727415 29744 raft_consensus.cc:348] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.727654 29744 raft_consensus.cc:374] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:51.727706 29744 raft_consensus.cc:729] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e73a50fd61462f84ce3878f864fd5e, State: Initialized, Role: FOLLOWER
I20260927 10:48:51.728150 29744 consensus_queue.cc:260] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.728492 29746 raft_consensus.cc:348] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.728490 29745 raft_consensus.cc:348] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.728739 29745 raft_consensus.cc:374] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:51.728739 29746 raft_consensus.cc:374] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:51.728803 29745 raft_consensus.cc:729] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d66da5ffef4682820b98daffeaba7c, State: Initialized, Role: FOLLOWER
I20260927 10:48:51.728803 29746 raft_consensus.cc:729] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90f24dcadb24a20872a50b9b9994a13, State: Initialized, Role: FOLLOWER
I20260927 10:48:51.728822 29465 heartbeater.cc:503] Master 127.10.96.254:39793 was elected leader, sending a full tablet report...
I20260927 10:48:51.729229 29746 consensus_queue.cc:260] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.729230 29745 consensus_queue.cc:260] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.730195 29745 ts_tablet_manager.cc:1428] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 10:48:51.730195 29746 ts_tablet_manager.cc:1428] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 10:48:51.730513 29596 heartbeater.cc:503] Master 127.10.96.254:39793 was elected leader, sending a full tablet report...
I20260927 10:48:51.730531 29744 ts_tablet_manager.cc:1428] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:48:51.731482 29727 heartbeater.cc:503] Master 127.10.96.254:39793 was elected leader, sending a full tablet report...
W20260927 10:48:51.784704 29466 tablet.cc:2369] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:51.817875 29752 raft_consensus.cc:482] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:51.818156 29752 raft_consensus.cc:504] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.819443 29752 leader_election.cc:288] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07e73a50fd61462f84ce3878f864fd5e (127.10.96.193:39427), 89d66da5ffef4682820b98daffeaba7c (127.10.96.194:36581)
I20260927 10:48:51.823717 29743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.819882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:45949 (local address 127.10.96.194:36581)
0927 10:48:51.820021 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.820028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.820287 (+ 259us) server_negotiation.cc:407] Connection header received
0927 10:48:51.820378 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.820381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.820429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.820511 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:51.820839 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.821368 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.822699 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.823097 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.823222 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.823324 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.823498 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.823580 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:48:51.823737 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.819638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39427 (local address 127.10.96.195:54835)
0927 10:48:51.819877 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.819896 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.820023 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.820325 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.820328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.820352 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:51.820507 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.820516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.821372 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.821376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.822480 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.822490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.823262 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.823265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.823424 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.823451 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.823532 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.823594 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":23}
I20260927 10:48:51.823765 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.819767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:54835 (local address 127.10.96.193:39427)
0927 10:48:51.819903 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.819909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.820061 (+ 152us) server_negotiation.cc:407] Connection header received
0927 10:48:51.820129 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.820132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.820202 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.820289 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:51.820634 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.821261 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.822640 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.823107 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.823253 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.823313 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.823566 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.823652 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:48:51.823764 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.819781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:36581 (local address 127.10.96.195:45949)
0927 10:48:51.820164 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.820178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:51.820292 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:51.820538 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:51.820541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:51.820555 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:51.820711 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.820718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.821517 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.821521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.822582 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:51.822597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.823220 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.823224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:51.823324 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.823400 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 10:48:51.823470 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 10:48:51.823566 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":19,"thread_start_us":118,"threads_started":1}
I20260927 10:48:51.824404 29550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83301c220415436fb46872b40da50eee" candidate_uuid: "a90f24dcadb24a20872a50b9b9994a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d66da5ffef4682820b98daffeaba7c" is_pre_election: true
I20260927 10:48:51.824496 29419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83301c220415436fb46872b40da50eee" candidate_uuid: "a90f24dcadb24a20872a50b9b9994a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e73a50fd61462f84ce3878f864fd5e" is_pre_election: true
I20260927 10:48:51.824710 29550 raft_consensus.cc:2454] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90f24dcadb24a20872a50b9b9994a13 in term 0.
I20260927 10:48:51.824844 29419 raft_consensus.cc:2454] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90f24dcadb24a20872a50b9b9994a13 in term 0.
I20260927 10:48:51.825207 29616 leader_election.cc:302] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d66da5ffef4682820b98daffeaba7c, a90f24dcadb24a20872a50b9b9994a13; no voters:
I20260927 10:48:51.825513 29752 raft_consensus.cc:2781] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:51.825604 29752 raft_consensus.cc:482] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:51.825640 29752 raft_consensus.cc:3037] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:51.826717 29752 raft_consensus.cc:504] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.827092 29752 leader_election.cc:288] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [CANDIDATE]: Term 1 election: Requested vote from peers 07e73a50fd61462f84ce3878f864fd5e (127.10.96.193:39427), 89d66da5ffef4682820b98daffeaba7c (127.10.96.194:36581)
I20260927 10:48:51.827534 29550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83301c220415436fb46872b40da50eee" candidate_uuid: "a90f24dcadb24a20872a50b9b9994a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89d66da5ffef4682820b98daffeaba7c"
I20260927 10:48:51.827612 29419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83301c220415436fb46872b40da50eee" candidate_uuid: "a90f24dcadb24a20872a50b9b9994a13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e73a50fd61462f84ce3878f864fd5e"
I20260927 10:48:51.827662 29550 raft_consensus.cc:3037] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:51.827766 29419 raft_consensus.cc:3037] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:51.828773 29550 raft_consensus.cc:2454] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90f24dcadb24a20872a50b9b9994a13 in term 1.
I20260927 10:48:51.828794 29419 raft_consensus.cc:2454] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90f24dcadb24a20872a50b9b9994a13 in term 1.
I20260927 10:48:51.829085 29616 leader_election.cc:302] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89d66da5ffef4682820b98daffeaba7c, a90f24dcadb24a20872a50b9b9994a13; no voters:
I20260927 10:48:51.829329 29752 raft_consensus.cc:2781] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:51.829589 29752 raft_consensus.cc:686] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 1 LEADER]: Becoming Leader. State: Replica: a90f24dcadb24a20872a50b9b9994a13, State: Running, Role: LEADER
I20260927 10:48:51.829908 29752 consensus_queue.cc:237] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } }
I20260927 10:48:51.833264 29276 catalog_manager.cc:5495] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 reported cstate change: term changed from 0 to 1, leader changed from <none> to a90f24dcadb24a20872a50b9b9994a13 (127.10.96.195). New cstate: current_term: 1 leader_uuid: "a90f24dcadb24a20872a50b9b9994a13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90f24dcadb24a20872a50b9b9994a13" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 41961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 } health_report { overall_health: UNKNOWN } } }
I20260927 10:48:51.860965 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.857272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.10.96.195:41961)
0927 10:48:51.857414 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:51.857418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:51.857434 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:48:51.857614 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:51.857617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:51.857677 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:51.857799 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:51.858054 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.858563 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.859444 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:51.859670 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:51.859782 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:51.860106 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:51.860529 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:51.860702 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:51.860761 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 10:48:51.860796 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:48:51.867551 29419 raft_consensus.cc:1261] T 83301c220415436fb46872b40da50eee P 07e73a50fd61462f84ce3878f864fd5e [term 1 FOLLOWER]: Refusing update from remote peer a90f24dcadb24a20872a50b9b9994a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:51.867604 29550 raft_consensus.cc:1261] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 1 FOLLOWER]: Refusing update from remote peer a90f24dcadb24a20872a50b9b9994a13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:51.868285 29752 consensus_queue.cc:1035] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07e73a50fd61462f84ce3878f864fd5e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:51.868506 29754 consensus_queue.cc:1035] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89d66da5ffef4682820b98daffeaba7c" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 36581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:51.877140 29759 mvcc.cc:205] Tried to move back new op lower bound from 7333913116121235456 to 7333913115978022912. Current Snapshot: MvccSnapshot[applied={T|T < 7333913116121235456}]
I20260927 10:48:51.885582 29277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:51.885762 29277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:51.888285 29277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28660fb6583e4f5ba273328de021225d: Sending DeleteTablet for 3 replicas of tablet 83301c220415436fb46872b40da50eee
I20260927 10:48:51.889112 29661 tablet_service.cc:1514] Processing DeleteTablet for tablet 83301c220415436fb46872b40da50eee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:51 UTC) from {username='slave'} at 127.0.0.1:53992
I20260927 10:48:51.889202 29530 tablet_service.cc:1514] Processing DeleteTablet for tablet 83301c220415436fb46872b40da50eee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:51 UTC) from {username='slave'} at 127.0.0.1:59784
I20260927 10:48:51.889679 29399 tablet_service.cc:1514] Processing DeleteTablet for tablet 83301c220415436fb46872b40da50eee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:51 UTC) from {username='slave'} at 127.0.0.1:58766
I20260927 10:48:51.889817 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29337
I20260927 10:48:51.894362 29768 tablet_replica.cc:330] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: stopping tablet replica
I20260927 10:48:51.894737 29768 raft_consensus.cc:2229] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:51.889647 29767 tablet_replica.cc:330] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: stopping tablet replica
I20260927 10:48:51.895088 29768 raft_consensus.cc:2258] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:51.895231 29767 raft_consensus.cc:2229] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 1 LEADER]: Raft consensus shutting down.
I20260927 10:48:51.895634 29767 raft_consensus.cc:2258] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 10:48:51.896572 29261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:39427 (error 108)
I20260927 10:48:51.897125 29767 ts_tablet_manager.cc:1905] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:51.897135 29768 ts_tablet_manager.cc:1905] T 83301c220415436fb46872b40da50eee P 89d66da5ffef4682820b98daffeaba7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:51.897347 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29468
I20260927 10:48:51.900717 29767 ts_tablet_manager.cc:1918] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:51.900820 29767 log.cc:1196] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/83301c220415436fb46872b40da50eee
I20260927 10:48:51.901147 29767 ts_tablet_manager.cc:1939] T 83301c220415436fb46872b40da50eee P a90f24dcadb24a20872a50b9b9994a13: Deleting consensus metadata
I20260927 10:48:51.902631 29262 catalog_manager.cc:4841] TS a90f24dcadb24a20872a50b9b9994a13 (127.10.96.195:41961): tablet 83301c220415436fb46872b40da50eee (table test-table [id=0cc1ffe99d4c4058a1aa1de895157528]) successfully deleted
I20260927 10:48:51.903110 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.902690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39427 (local address 127.0.0.1:58774)
0927 10:48:51.902878 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.902993 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 10:48:51.903308 29261 catalog_manager.cc:4584] TS 07e73a50fd61462f84ce3878f864fd5e (127.10.96.193:39427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83301c220415436fb46872b40da50eee: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39427: connect: Connection refused (error 111)
I20260927 10:48:51.910460 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29599
I20260927 10:48:51.913304 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:51.910287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:36581 (local address 127.0.0.1:59800)
0927 10:48:51.913112 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:51.913183 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:36581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2710}
I20260927 10:48:51.920260 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29246
2026-09-27T10:48:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1091 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:51 dist-test-slave-wmg0 krb5kdc[29772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:51 dist-test-slave-wmg0 krb5kdc[29772](info): set up 2 sockets
Sep 27 10:48:51 dist-test-slave-wmg0 krb5kdc[29772](info): commencing operation
krb5kdc: starting...
W20260927 10:48:52.969445 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:48:53.092399 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:42367
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:42367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:53.202992 29785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:53.203315 29785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:53.203392 29785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:53.207496 29785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:53.207581 29785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:53.207620 29785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:53.207677 29785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:53.207713 29785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:53.213191 29785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:42367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:42367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:53.214391 29785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:53.215406 29785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:53.223872 29790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.223876 29793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.223872 29791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:53.224375 29785 server_base.cc:1034] running on GCE node
I20260927 10:48:53.224970 29785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:53.226154 29785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:53.227326 29785 hybrid_clock.cc:648] HybridClock initialized: now 1790506133227311 us; error 25 us; skew 500 ppm
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:53.230311 29785 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:48:53.231233 29785 webserver.cc:460] Webserver started at http://127.10.96.254:41043/ using document root <none> and password file <none>
I20260927 10:48:53.231806 29785 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:53.231868 29785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:53.232084 29785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:53.234278 29785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "f0585fcb7eb5657e39b9223659afdbc6"
server_key_iv: "2a7f7e3573f9a1177c846523b5ff958f"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.234860 29785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "f0585fcb7eb5657e39b9223659afdbc6"
server_key_iv: "2a7f7e3573f9a1177c846523b5ff958f"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.239094 29785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:53.241803 29800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.243165 29785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:48:53.243328 29785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "f0585fcb7eb5657e39b9223659afdbc6"
server_key_iv: "2a7f7e3573f9a1177c846523b5ff958f"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.243453 29785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:53.256023 29785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:53.259580 29785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:53.259809 29785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:53.268488 29785 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:42367
I20260927 10:48:53.268536 29852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:42367 every 8 connection(s)
I20260927 10:48:53.269901 29785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:53.278813 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29785
I20260927 10:48:53.278985 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:53.279287 10627 external_mini_cluster.cc:1428] Setting key da7275e1549f4f541393081c7385f1ec
I20260927 10:48:53.279592 29853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:53.287153 29853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe: Bootstrap starting.
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:53.289619 29853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:53.290592 29853 log.cc:824] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:53.293061 29853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe: No bootstrap required, opened a new log
I20260927 10:48:53.295409 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:53.280721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.10.96.254:42367)
0927 10:48:53.281191 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:53.281213 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:53.281258 (+ 45us) server_negotiation.cc:407] Connection header received
0927 10:48:53.282128 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:53.282155 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:53.282623 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:53.283038 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:53.283768 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.285217 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.286184 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.286863 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.287032 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:53.289709 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:53.289716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:53.289732 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:53.289768 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:53.291716 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:53.292195 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:53.292205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:53.292212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:53.292348 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:53.292669 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:53.292672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:53.292673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:53.293194 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:53.293382 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:53.293748 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 10:48:53.294061 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":137,"threads_started":1}
I20260927 10:48:53.297633 29853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42367 } }
I20260927 10:48:53.297875 29853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:53.297935 29853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c253d01cb9e4d8fa8f94409fbe1a9fe, State: Initialized, Role: FOLLOWER
I20260927 10:48:53.298465 29853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42367 } }
I20260927 10:48:53.298656 29853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:53.298722 29853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:53.298823 29853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:53.299827 29853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42367 } }
I20260927 10:48:53.300197 29853 leader_election.cc:302] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c253d01cb9e4d8fa8f94409fbe1a9fe; no voters:
I20260927 10:48:53.300537 29853 leader_election.cc:288] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:53.300805 29858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:53.301069 29858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [term 1 LEADER]: Becoming Leader. State: Replica: 1c253d01cb9e4d8fa8f94409fbe1a9fe, State: Running, Role: LEADER
I20260927 10:48:53.301436 29858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42367 } }
I20260927 10:48:53.301934 29853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:53.303119 29859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42367 } } }
I20260927 10:48:53.303248 29859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:53.303584 29860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c253d01cb9e4d8fa8f94409fbe1a9fe. Latest consensus state: current_term: 1 leader_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c253d01cb9e4d8fa8f94409fbe1a9fe" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42367 } } }
I20260927 10:48:53.303678 29860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:53.304001 29867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:53.306784 29867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:53.312459 29867 catalog_manager.cc:1347] Generated new cluster ID: 5b596364cf474e5badad731a0fe37a6d
I20260927 10:48:53.312530 29867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:53.318866 29867 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:53.319876 29867 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:53.327651 29867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe: Generated new TSK 0
I20260927 10:48:53.328459 29867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
Entry for principal kudu/127.10.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab.
I20260927 10:48:53.360616 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42367
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:53.475014 29879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:53.475283 29879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:53.475327 29879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:53.479470 29879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:53.479542 29879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:53.479645 29879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:53.484935 29879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.193.keytab
--principal=kudu/127.10.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:53.486066 29879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:53.487135 29879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:53.496708 29885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.496694 29887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.496701 29884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:53.497105 29879 server_base.cc:1034] running on GCE node
I20260927 10:48:53.497576 29879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:53.498216 29879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:53.499393 29879 hybrid_clock.cc:648] HybridClock initialized: now 1790506133499367 us; error 57 us; skew 500 ppm
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:53.502386 29879 init.cc:377] Logged in from keytab as kudu/127.10.96.193@KRBTEST.COM (short username kudu)
I20260927 10:48:53.503441 29879 webserver.cc:460] Webserver started at http://127.10.96.193:39507/ using document root <none> and password file <none>
I20260927 10:48:53.503971 29879 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:53.504055 29879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:53.504276 29879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:53.506333 29879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "2248bd8aff48423ea33cf63abcc748e4"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "cbadf641ce3be61fa0b52f68761697dd"
server_key_iv: "6d4eb747ff8ba33e468eb997574f9a9c"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.506913 29879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "2248bd8aff48423ea33cf63abcc748e4"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "cbadf641ce3be61fa0b52f68761697dd"
server_key_iv: "6d4eb747ff8ba33e468eb997574f9a9c"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.510859 29879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:48:53.513577 29894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.514835 29879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:53.514962 29879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "2248bd8aff48423ea33cf63abcc748e4"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "cbadf641ce3be61fa0b52f68761697dd"
server_key_iv: "6d4eb747ff8ba33e468eb997574f9a9c"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.515074 29879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:53.526171 29879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:53.529528 29879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:53.529736 29879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:53.530344 29879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:53.531209 29879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:53.531258 29879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.531301 29879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:53.531315 29879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.540202 29879 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39831
I20260927 10:48:53.540221 30007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39831 every 8 connection(s)
I20260927 10:48:53.541110 29879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:53.547631 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 29879
I20260927 10:48:53.547771 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:48:53.548058 10627 external_mini_cluster.cc:1428] Setting key e187dc6be411cc358a9f05425c3cbdf7
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.193@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:53.556347 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:53.543066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:47093 (local address 127.10.96.254:42367)
0927 10:48:53.543210 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:53.543214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:53.544098 (+ 884us) server_negotiation.cc:407] Connection header received
0927 10:48:53.545516 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:53.545520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:53.545580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:53.545684 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:53.547023 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.547684 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.549182 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.549564 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.549708 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:53.552283 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:53.552287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:53.552289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:53.552312 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:53.553897 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:53.554560 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:53.554563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:53.554564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:53.554642 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:53.555092 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:53.555095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:53.555096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:53.555280 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:53.555349 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:53.556094 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 10:48:53.556221 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:48:53.557682 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:53.543465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42367 (local address 127.10.96.193:47093)
0927 10:48:53.543920 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:53.543988 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:53.545266 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:53.545881 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:53.545888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:53.546258 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:53.546833 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:53.546848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.547873 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.547877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:53.549066 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:53.549077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.549765 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.549768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:53.549919 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:53.550576 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:53.550592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:53.552092 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:53.554015 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:53.554023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:53.554041 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:53.554447 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:53.554746 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:53.554752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:53.554774 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:53.554986 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:53.555405 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:53.555415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:53.555811 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 10:48:53.556067 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 10:48:53.556477 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.10.96.194@KRBTEST.COM; defaulting to no policy
I20260927 10:48:53.559089 30008 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42367
I20260927 10:48:53.559374 30008 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:53.559914 30008 heartbeater.cc:511] Master 127.10.96.254:42367 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.194@KRBTEST.COM" created.
I20260927 10:48:53.561975 29808 ts_manager.cc:194] Registered new tserver with Master: 2248bd8aff48423ea33cf63abcc748e4 (127.10.96.193:39831)
I20260927 10:48:53.563601 29808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.193@KRBTEST.COM'} at 127.10.96.193:47093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
Entry for principal kudu/127.10.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab.
I20260927 10:48:53.576711 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42367
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:53.689764 30013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:53.690120 30013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:53.690208 30013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:53.694351 30013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:53.694459 30013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:53.694556 30013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:53.700103 30013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.194.keytab
--principal=kudu/127.10.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:53.701309 30013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:53.702576 30013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:53.712587 30019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.712601 30018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.712648 30021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:53.712837 30013 server_base.cc:1034] running on GCE node
I20260927 10:48:53.713341 30013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:53.714036 30013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:53.715256 30013 hybrid_clock.cc:648] HybridClock initialized: now 1790506133715214 us; error 59 us; skew 500 ppm
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:53.718327 30013 init.cc:377] Logged in from keytab as kudu/127.10.96.194@KRBTEST.COM (short username kudu)
I20260927 10:48:53.719321 30013 webserver.cc:460] Webserver started at http://127.10.96.194:40257/ using document root <none> and password file <none>
I20260927 10:48:53.719898 30013 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:53.719990 30013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:53.720227 30013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:53.722374 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "a2835f69f1d24939bfa432fdbd2f4803"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "f2eed5d711d2593e35f5b9f0c60ad26e"
server_key_iv: "9535c5108c9344a7158de6348e799a84"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.722983 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "a2835f69f1d24939bfa432fdbd2f4803"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "f2eed5d711d2593e35f5b9f0c60ad26e"
server_key_iv: "9535c5108c9344a7158de6348e799a84"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.726681 30013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:53.729027 30028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.730276 30013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:53.730414 30013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "a2835f69f1d24939bfa432fdbd2f4803"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "f2eed5d711d2593e35f5b9f0c60ad26e"
server_key_iv: "9535c5108c9344a7158de6348e799a84"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.730526 30013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:53.767864 30013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:53.771534 30013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:53.771754 30013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:53.772392 30013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:53.773262 30013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:53.773335 30013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.773407 30013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:53.773458 30013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.782428 30013 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:43973
I20260927 10:48:53.782456 30141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:43973 every 8 connection(s)
I20260927 10:48:53.783354 30013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:53.783943 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30013
I20260927 10:48:53.784112 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:48:53.784422 10627 external_mini_cluster.cc:1428] Setting key d8c4fffd3bf873141fdf93daec20f844
WARNING: no policy specified for kudu/127.10.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.195@KRBTEST.COM" created.
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.194@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:53.802412 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:53.785391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:36897 (local address 127.10.96.254:42367)
0927 10:48:53.785569 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:53.785574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:53.786525 (+ 951us) server_negotiation.cc:407] Connection header received
0927 10:48:53.788217 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:53.788220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:53.788265 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:53.788337 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:53.790030 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.790592 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.792469 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.792694 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.792777 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:53.798422 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:53.798426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:53.798428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:53.798459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:53.800051 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:53.800731 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:53.800734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:53.800736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:53.800813 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:53.801175 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:53.801179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:53.801180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:53.801373 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:53.801513 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:53.802109 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 10:48:53.802244 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 10:48:53.803449 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:53.785876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42367 (local address 127.10.96.194:36897)
0927 10:48:53.786353 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:53.786405 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:53.787973 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:53.788574 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:53.788586 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:53.789120 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:53.789861 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:53.789884 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.790731 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.790736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:53.792343 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:53.792355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:53.792862 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:53.792869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:53.793093 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:53.793801 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:53.793822 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:53.798183 (+ 4361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:53.800300 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:53.800307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:53.800325 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:53.800608 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:53.800908 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:53.800912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:53.800914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:53.801071 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:53.801506 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:53.801513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:53.801785 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 10:48:53.802072 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 10:48:53.802433 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260927 10:48:53.804715 30142 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42367
I20260927 10:48:53.805008 30142 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:53.805680 30142 heartbeater.cc:511] Master 127.10.96.254:42367 requested a full tablet report, sending...
I20260927 10:48:53.806830 29808 ts_manager.cc:194] Registered new tserver with Master: a2835f69f1d24939bfa432fdbd2f4803 (127.10.96.194:43973)
I20260927 10:48:53.807428 29808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.194@KRBTEST.COM'} at 127.10.96.194:36897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
Entry for principal kudu/127.10.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab.
I20260927 10:48:53.812479 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42367
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:53.926928 30147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:53.927225 30147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:53.927271 30147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:53.931932 30147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:53.932039 30147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:53.932116 30147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:53.938138 30147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.195.keytab
--principal=kudu/127.10.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:53.939219 30147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:53.940204 30147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:53.949921 30153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:53.949990 30155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:53.950178 30147 server_base.cc:1034] running on GCE node
W20260927 10:48:53.950204 30152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:53.950670 30147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:53.951328 30147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:53.952500 30147 hybrid_clock.cc:648] HybridClock initialized: now 1790506133952484 us; error 28 us; skew 500 ppm
Sep 27 10:48:53 dist-test-slave-wmg0 krb5kdc[29772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:53.955406 30147 init.cc:377] Logged in from keytab as kudu/127.10.96.195@KRBTEST.COM (short username kudu)
I20260927 10:48:53.956327 30147 webserver.cc:460] Webserver started at http://127.10.96.195:35831/ using document root <none> and password file <none>
I20260927 10:48:53.956905 30147 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:53.957010 30147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:53.957242 30147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:53.959460 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "13f89919e8c94afeb781d89699ef52d5"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "29511eb789137ad8741d2244e4190aca"
server_key_iv: "885f7345c26eca3cf9d7433102355815"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.960047 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "13f89919e8c94afeb781d89699ef52d5"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "29511eb789137ad8741d2244e4190aca"
server_key_iv: "885f7345c26eca3cf9d7433102355815"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.963804 30147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:48:53.966256 30162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.967486 30147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:53.967618 30147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "13f89919e8c94afeb781d89699ef52d5"
format_stamp: "Formatted at 2026-09-27 10:48:53 on dist-test-slave-wmg0"
server_key: "29511eb789137ad8741d2244e4190aca"
server_key_iv: "885f7345c26eca3cf9d7433102355815"
server_key_version: "encryptionkey@0"
I20260927 10:48:53.967729 30147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:53.988981 30147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:53.992246 30147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:53.992412 30147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:53.992947 30147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:53.993803 30147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:53.993852 30147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:53.993892 30147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:53.993908 30147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.003265 30147 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:35749
I20260927 10:48:54.003353 30275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:35749 every 8 connection(s)
I20260927 10:48:54.004204 30147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:54.009384 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30147
I20260927 10:48:54.009555 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:48:54.009858 10627 external_mini_cluster.cc:1428] Setting key 037b349da33950f25e37086ece3320e0
Sep 27 10:48:54 dist-test-slave-wmg0 krb5kdc[29772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506133, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.195@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:48:54.018756 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.006151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:57455 (local address 127.10.96.254:42367)
0927 10:48:54.006322 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.006327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.007176 (+ 849us) server_negotiation.cc:407] Connection header received
0927 10:48:54.008392 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.008394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.008439 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.008525 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:54.010032 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.010649 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.012256 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.012546 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.012650 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.015376 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:54.015379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:54.015382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:54.015405 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:54.016924 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:54.017371 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:54.017374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:54.017375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:54.017444 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:54.017771 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:54.017774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:54.017775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:54.017964 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:54.018088 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.018524 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.018636 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:48:54.019717 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.006555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42367 (local address 127.10.96.195:57455)
0927 10:48:54.007022 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.007065 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.008203 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.008687 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.008694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.009066 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:54.009843 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.009863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.010819 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.010825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.012156 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.012165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.012716 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.012719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.012873 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.013628 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:48:54.013644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:54.015220 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:54.017035 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:54.017039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:54.017051 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:54.017293 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:54.017562 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:48:54.017565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:48:54.017566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:48:54.017698 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:48:54.018092 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:54.018098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:54.018327 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.018485 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.018776 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":112,"threads_started":1}
I20260927 10:48:54.020864 30276 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42367
I20260927 10:48:54.021091 30276 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:54.021628 30276 heartbeater.cc:511] Master 127.10.96.254:42367 requested a full tablet report, sending...
I20260927 10:48:54.022740 29808 ts_manager.cc:194] Registered new tserver with Master: 13f89919e8c94afeb781d89699ef52d5 (127.10.96.195:35749)
I20260927 10:48:54.023483 29808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.96.195@KRBTEST.COM'} at 127.10.96.195:57455
I20260927 10:48:54.023830 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:54.032948 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.025903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.10.96.254:42367)
0927 10:48:54.026082 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.026087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.026289 (+ 202us) server_negotiation.cc:407] Connection header received
0927 10:48:54.026382 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.026384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.026422 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.026500 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:54.027154 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.027647 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.028452 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.028673 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.028750 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.030233 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:54.030237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:54.030239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:54.030261 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:54.031545 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:54.031924 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:54.031926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:54.031928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:54.032009 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:54.032236 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:54.032238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:54.032239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:54.032437 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:54.032545 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.032740 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.032833 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 10:48:54.036160 29808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:54.039204 29808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:54.053272 30287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.047704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:35749 (local address 127.0.0.1:35396)
0927 10:48:54.048114 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.048168 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.048299 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.049016 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.049022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.049044 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:54.049150 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.049155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.050565 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.050569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.051521 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.051530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.052630 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.052633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.052854 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.052904 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.053004 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.053115 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":69,"thread_start_us":102,"threads_started":1}
I20260927 10:48:54.053280 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.047855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.10.96.195:35749)
0927 10:48:54.048333 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.048344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.048369 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:48:54.048442 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.048447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.048672 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.048837 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:54.049268 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.050438 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.051623 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.052496 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.052796 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.052982 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.053039 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.053135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":151,"threads_started":1}
I20260927 10:48:54.053908 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.048112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:43973 (local address 127.0.0.1:41082)
0927 10:48:54.048687 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.048701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.048790 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.049697 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.049700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.049716 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:54.049828 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.049836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.051399 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.051403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.052439 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.052448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.053622 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.053627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.053704 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.053723 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.053762 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.053809 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":50,"thread_start_us":84,"threads_started":1}
I20260927 10:48:54.054648 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.048882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.10.96.194:43973)
0927 10:48:54.049217 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.049228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.049250 (+ 22us) server_negotiation.cc:407] Connection header received
0927 10:48:54.049323 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.049328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.049536 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.049994 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:54.050134 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.051258 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.052589 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.053493 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.054230 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.054376 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.054426 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.054516 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":148,"threads_started":1}
I20260927 10:48:54.054786 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.048469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39831 (local address 127.0.0.1:37946)
0927 10:48:54.049379 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.049392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.049497 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.050087 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.050090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.050105 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:54.050193 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.050198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.051545 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.051550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.052789 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.052798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.054389 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.054392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.054497 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.054519 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.054591 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.054661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":29,"thread_start_us":92,"threads_started":1}
I20260927 10:48:54.054836 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.048623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.10.96.193:39831)
0927 10:48:54.048985 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.049001 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.049594 (+ 593us) server_negotiation.cc:407] Connection header received
0927 10:48:54.049671 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.049675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.049832 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.049956 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:54.050318 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.051401 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.052923 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.054253 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.054392 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.054535 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.054619 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.054707 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":72,"threads_started":1}
I20260927 10:48:54.056668 30210 tablet_service.cc:1467] Processing CreateTablet for tablet a4723f61ba48414b9cd52cf8beeefe9d (DEFAULT_TABLE table=test-table [id=3a94c423201c4482b5f6882aeb3285d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:54.057037 29942 tablet_service.cc:1467] Processing CreateTablet for tablet a4723f61ba48414b9cd52cf8beeefe9d (DEFAULT_TABLE table=test-table [id=3a94c423201c4482b5f6882aeb3285d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:54.057276 30076 tablet_service.cc:1467] Processing CreateTablet for tablet a4723f61ba48414b9cd52cf8beeefe9d (DEFAULT_TABLE table=test-table [id=3a94c423201c4482b5f6882aeb3285d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:54.058030 29942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4723f61ba48414b9cd52cf8beeefe9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:54.058269 30076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4723f61ba48414b9cd52cf8beeefe9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:54.058323 30210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4723f61ba48414b9cd52cf8beeefe9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:54.064345 30293 tablet_bootstrap.cc:492] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: Bootstrap starting.
I20260927 10:48:54.065747 30295 tablet_bootstrap.cc:492] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Bootstrap starting.
I20260927 10:48:54.065747 30294 tablet_bootstrap.cc:492] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Bootstrap starting.
I20260927 10:48:54.067624 30293 tablet_bootstrap.cc:654] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:54.067938 30295 tablet_bootstrap.cc:654] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:54.067938 30294 tablet_bootstrap.cc:654] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:54.068524 30293 log.cc:824] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:54.069375 30295 log.cc:824] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:54.069375 30294 log.cc:824] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:54.070757 30293 tablet_bootstrap.cc:492] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: No bootstrap required, opened a new log
I20260927 10:48:54.070956 30293 ts_tablet_manager.cc:1397] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 10:48:54.071293 30294 tablet_bootstrap.cc:492] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: No bootstrap required, opened a new log
I20260927 10:48:54.071460 30294 ts_tablet_manager.cc:1397] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 10:48:54.071738 30295 tablet_bootstrap.cc:492] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: No bootstrap required, opened a new log
I20260927 10:48:54.071904 30295 ts_tablet_manager.cc:1397] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260927 10:48:54.075106 30293 raft_consensus.cc:348] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.075371 30293 raft_consensus.cc:374] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:54.075426 30293 raft_consensus.cc:729] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2248bd8aff48423ea33cf63abcc748e4, State: Initialized, Role: FOLLOWER
I20260927 10:48:54.075376 30294 raft_consensus.cc:348] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.075652 30294 raft_consensus.cc:374] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:54.075697 30294 raft_consensus.cc:729] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2835f69f1d24939bfa432fdbd2f4803, State: Initialized, Role: FOLLOWER
I20260927 10:48:54.075556 30295 raft_consensus.cc:348] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.075783 30295 raft_consensus.cc:374] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:54.075834 30295 raft_consensus.cc:729] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f89919e8c94afeb781d89699ef52d5, State: Initialized, Role: FOLLOWER
I20260927 10:48:54.075924 30293 consensus_queue.cc:260] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.076161 30294 consensus_queue.cc:260] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.076195 30295 consensus_queue.cc:260] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.076671 30008 heartbeater.cc:503] Master 127.10.96.254:42367 was elected leader, sending a full tablet report...
I20260927 10:48:54.076982 30293 ts_tablet_manager.cc:1428] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:48:54.077087 30295 ts_tablet_manager.cc:1428] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 10:48:54.077087 30294 ts_tablet_manager.cc:1428] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 10:48:54.077418 30142 heartbeater.cc:503] Master 127.10.96.254:42367 was elected leader, sending a full tablet report...
I20260927 10:48:54.078318 30276 heartbeater.cc:503] Master 127.10.96.254:42367 was elected leader, sending a full tablet report...
I20260927 10:48:54.209564 30300 raft_consensus.cc:482] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:54.209829 30300 raft_consensus.cc:504] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.211143 30300 leader_election.cc:288] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2248bd8aff48423ea33cf63abcc748e4 (127.10.96.193:39831), a2835f69f1d24939bfa432fdbd2f4803 (127.10.96.194:43973)
I20260927 10:48:54.214885 30292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.211654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:49383 (local address 127.10.96.194:43973)
0927 10:48:54.211793 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.211798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.211813 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:48:54.211871 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.211873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.211913 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.212008 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:54.212278 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.212790 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.213918 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.214320 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.214460 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.214514 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.214696 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.214763 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:48:54.214885 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.211423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:43973 (local address 127.10.96.195:49383)
0927 10:48:54.211579 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.211596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.211706 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.212019 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.212022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.212040 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:54.212157 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.212163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.212903 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.212909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.213794 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.213803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.214443 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.214447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.214561 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.214583 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.214680 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.214757 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 10:48:54.214996 30291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.211570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:47103 (local address 127.10.96.193:39831)
0927 10:48:54.211816 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.211821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.211931 (+ 110us) server_negotiation.cc:407] Connection header received
0927 10:48:54.212076 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.212078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.212131 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.212216 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:54.212473 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.212991 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.214074 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.214469 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.214593 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.214639 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.214800 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.214863 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 10:48:54.215042 30302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.211436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39831 (local address 127.10.96.195:47103)
0927 10:48:54.211822 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.211836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.211969 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.212224 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.212227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.212241 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:54.212367 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.212372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.213105 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.213111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.213959 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.213967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.214582 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.214586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.214680 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.214704 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.214786 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.214851 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":27,"thread_start_us":117,"threads_started":1}
I20260927 10:48:54.215580 29962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4723f61ba48414b9cd52cf8beeefe9d" candidate_uuid: "13f89919e8c94afeb781d89699ef52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2248bd8aff48423ea33cf63abcc748e4" is_pre_election: true
I20260927 10:48:54.215639 30096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4723f61ba48414b9cd52cf8beeefe9d" candidate_uuid: "13f89919e8c94afeb781d89699ef52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2835f69f1d24939bfa432fdbd2f4803" is_pre_election: true
I20260927 10:48:54.215883 29962 raft_consensus.cc:2454] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13f89919e8c94afeb781d89699ef52d5 in term 0.
I20260927 10:48:54.215978 30096 raft_consensus.cc:2454] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13f89919e8c94afeb781d89699ef52d5 in term 0.
I20260927 10:48:54.216355 30165 leader_election.cc:302] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f89919e8c94afeb781d89699ef52d5, 2248bd8aff48423ea33cf63abcc748e4; no voters:
I20260927 10:48:54.216651 30300 raft_consensus.cc:2781] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:54.216734 30300 raft_consensus.cc:482] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:54.216776 30300 raft_consensus.cc:3037] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:54.217885 30300 raft_consensus.cc:504] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.218227 30300 leader_election.cc:288] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [CANDIDATE]: Term 1 election: Requested vote from peers 2248bd8aff48423ea33cf63abcc748e4 (127.10.96.193:39831), a2835f69f1d24939bfa432fdbd2f4803 (127.10.96.194:43973)
I20260927 10:48:54.218621 29962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4723f61ba48414b9cd52cf8beeefe9d" candidate_uuid: "13f89919e8c94afeb781d89699ef52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2248bd8aff48423ea33cf63abcc748e4"
I20260927 10:48:54.218706 30096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4723f61ba48414b9cd52cf8beeefe9d" candidate_uuid: "13f89919e8c94afeb781d89699ef52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2835f69f1d24939bfa432fdbd2f4803"
I20260927 10:48:54.218743 29962 raft_consensus.cc:3037] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:54.218813 30096 raft_consensus.cc:3037] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:54.220094 30096 raft_consensus.cc:2454] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13f89919e8c94afeb781d89699ef52d5 in term 1.
I20260927 10:48:54.220094 29962 raft_consensus.cc:2454] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13f89919e8c94afeb781d89699ef52d5 in term 1.
I20260927 10:48:54.220449 30165 leader_election.cc:302] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f89919e8c94afeb781d89699ef52d5, 2248bd8aff48423ea33cf63abcc748e4; no voters:
I20260927 10:48:54.220639 30300 raft_consensus.cc:2781] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:54.220889 30300 raft_consensus.cc:686] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 1 LEADER]: Becoming Leader. State: Replica: 13f89919e8c94afeb781d89699ef52d5, State: Running, Role: LEADER
I20260927 10:48:54.221297 30300 consensus_queue.cc:237] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } }
I20260927 10:48:54.224586 29808 catalog_manager.cc:5495] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13f89919e8c94afeb781d89699ef52d5 (127.10.96.195). New cstate: current_term: 1 leader_uuid: "13f89919e8c94afeb781d89699ef52d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13f89919e8c94afeb781d89699ef52d5" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 35749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 } health_report { overall_health: UNKNOWN } } }
I20260927 10:48:54.246374 30288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.242648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.10.96.195:35749)
0927 10:48:54.242795 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.242801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.242817 (+ 16us) server_negotiation.cc:407] Connection header received
0927 10:48:54.242903 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.242905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.242969 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.243082 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:54.243334 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.243839 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.244774 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.244997 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.245111 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.245424 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:54.245937 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:54.246084 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.246121 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.246180 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 10:48:54.252552 30096 raft_consensus.cc:1261] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 1 FOLLOWER]: Refusing update from remote peer 13f89919e8c94afeb781d89699ef52d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:54.252552 29962 raft_consensus.cc:1261] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 1 FOLLOWER]: Refusing update from remote peer 13f89919e8c94afeb781d89699ef52d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:54.253289 30300 consensus_queue.cc:1035] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2248bd8aff48423ea33cf63abcc748e4" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:54.253517 30303 consensus_queue.cc:1035] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2835f69f1d24939bfa432fdbd2f4803" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 43973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 10:48:54.255096 30277 tablet.cc:2369] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:54.270414 29815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:54.270586 29815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:54.273065 29815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c253d01cb9e4d8fa8f94409fbe1a9fe: Sending DeleteTablet for 3 replicas of tablet a4723f61ba48414b9cd52cf8beeefe9d
I20260927 10:48:54.273955 30210 tablet_service.cc:1514] Processing DeleteTablet for tablet a4723f61ba48414b9cd52cf8beeefe9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:54 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:35396
I20260927 10:48:54.274097 29942 tablet_service.cc:1514] Processing DeleteTablet for tablet a4723f61ba48414b9cd52cf8beeefe9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:54 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:37946
I20260927 10:48:54.274480 30076 tablet_service.cc:1514] Processing DeleteTablet for tablet a4723f61ba48414b9cd52cf8beeefe9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:54 UTC) from {username='kudu', principal='kudu/127.10.96.254@KRBTEST.COM'} at 127.0.0.1:41082
I20260927 10:48:54.274724 30316 tablet_replica.cc:330] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4: stopping tablet replica
I20260927 10:48:54.275053 30316 raft_consensus.cc:2229] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:54.275388 30316 raft_consensus.cc:2258] T a4723f61ba48414b9cd52cf8beeefe9d P 2248bd8aff48423ea33cf63abcc748e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:54.275852 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29879
I20260927 10:48:54.278851 30317 tablet_replica.cc:330] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: stopping tablet replica
I20260927 10:48:54.279235 30317 raft_consensus.cc:2229] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:54.279606 30317 raft_consensus.cc:2258] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:54.281076 30317 ts_tablet_manager.cc:1905] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:54.283766 30317 ts_tablet_manager.cc:1918] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:54.283921 30317 log.cc:1196] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/a4723f61ba48414b9cd52cf8beeefe9d
I20260927 10:48:54.284328 30317 ts_tablet_manager.cc:1939] T a4723f61ba48414b9cd52cf8beeefe9d P a2835f69f1d24939bfa432fdbd2f4803: Deleting consensus metadata
I20260927 10:48:54.284498 30315 tablet_replica.cc:330] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: stopping tablet replica
I20260927 10:48:54.284821 30315 raft_consensus.cc:2229] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 1 LEADER]: Raft consensus shutting down.
W20260927 10:48:54.285044 29803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:39831 (error 108)
I20260927 10:48:54.285203 30315 raft_consensus.cc:2258] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:54.286033 29802 catalog_manager.cc:4841] TS a2835f69f1d24939bfa432fdbd2f4803 (127.10.96.194:43973): tablet a4723f61ba48414b9cd52cf8beeefe9d (table test-table [id=3a94c423201c4482b5f6882aeb3285d8]) successfully deleted
I20260927 10:48:54.286345 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30013
I20260927 10:48:54.286407 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.286114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39831 (local address 127.0.0.1:37960)
0927 10:48:54.286238 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.286307 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 10:48:54.287101 30315 ts_tablet_manager.cc:1905] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:54.289146 30315 ts_tablet_manager.cc:1918] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:54.289256 30315 log.cc:1196] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/a4723f61ba48414b9cd52cf8beeefe9d
I20260927 10:48:54.289618 30315 ts_tablet_manager.cc:1939] T a4723f61ba48414b9cd52cf8beeefe9d P 13f89919e8c94afeb781d89699ef52d5: Deleting consensus metadata
W20260927 10:48:54.289754 29803 catalog_manager.cc:4584] TS 2248bd8aff48423ea33cf63abcc748e4 (127.10.96.193:39831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4723f61ba48414b9cd52cf8beeefe9d: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39831: connect: Connection refused (error 111)
I20260927 10:48:54.290684 29803 catalog_manager.cc:4841] TS 13f89919e8c94afeb781d89699ef52d5 (127.10.96.195:35749): tablet a4723f61ba48414b9cd52cf8beeefe9d (table test-table [id=3a94c423201c4482b5f6882aeb3285d8]) successfully deleted
I20260927 10:48:54.295562 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30147
I20260927 10:48:54.300818 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 29785
2026-09-27T10:48:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2376 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T10:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:54Z Disabled control of system clock
I20260927 10:48:54.330365 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:42689
--webserver_interface=127.10.96.254
--webserver_port=0
--builtin_ntp_servers=127.10.96.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:48:54.439781 30322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:54.440052 30322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:54.440089 30322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:54.444209 30322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:54.444276 30322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:54.444291 30322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:54.444308 30322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:54.444324 30322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:54.449150 30322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:42689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:54.450311 30322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:54.451727 30322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:54.459868 30328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:54.459856 30327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:54.459885 30330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:54.460167 30322 server_base.cc:1034] running on GCE node
I20260927 10:48:54.460762 30322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:54.461898 30322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:54.463086 30322 hybrid_clock.cc:648] HybridClock initialized: now 1790506134463051 us; error 50 us; skew 500 ppm
I20260927 10:48:54.464845 30322 webserver.cc:460] Webserver started at http://127.10.96.254:44887/ using document root <none> and password file <none>
I20260927 10:48:54.465495 30322 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:54.465595 30322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:54.465824 30322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:54.468226 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "d0330df7aff847c8ba83b2c98fcb3f7b"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "960a2aea763e2696e81bf26e9a47ea36"
server_key_iv: "867688a2749747d1b1ad9610fe112050"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.468823 30322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "d0330df7aff847c8ba83b2c98fcb3f7b"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "960a2aea763e2696e81bf26e9a47ea36"
server_key_iv: "867688a2749747d1b1ad9610fe112050"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.472497 30322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:48:54.474929 30336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.476228 30322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:48:54.476397 30322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "d0330df7aff847c8ba83b2c98fcb3f7b"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "960a2aea763e2696e81bf26e9a47ea36"
server_key_iv: "867688a2749747d1b1ad9610fe112050"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.476518 30322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:54.490855 30322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:54.491535 30322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:54.491729 30322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:54.498739 30322 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:42689
I20260927 10:48:54.498751 30388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:42689 every 8 connection(s)
I20260927 10:48:54.499645 30322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:54.502758 30389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:54.506031 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30322
I20260927 10:48:54.506194 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:54.506484 10627 external_mini_cluster.cc:1428] Setting key bc2000c05c140cbcc231d844b06dc01c
I20260927 10:48:54.508541 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b: Bootstrap starting.
I20260927 10:48:54.510880 30389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:54.511618 30389 log.cc:824] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:54.513657 30389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b: No bootstrap required, opened a new log
I20260927 10:48:54.515838 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.507901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.10.96.254:42689)
0927 10:48:54.508529 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.508544 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.508579 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:48:54.509246 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.509262 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.509718 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.510087 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:54.510562 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.511648 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.512542 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.513022 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.513204 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.514156 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:54.514161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:54.514174 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:54.514559 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:54.514615 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:54.514635 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:54.514741 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.514905 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.515162 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260927 10:48:54.517285 30389 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42689 } }
I20260927 10:48:54.517565 30389 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:54.517652 30389 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0330df7aff847c8ba83b2c98fcb3f7b, State: Initialized, Role: FOLLOWER
I20260927 10:48:54.518111 30389 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42689 } }
I20260927 10:48:54.518287 30389 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:54.518355 30389 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:54.518448 30389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:54.519411 30389 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42689 } }
I20260927 10:48:54.519742 30389 leader_election.cc:302] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0330df7aff847c8ba83b2c98fcb3f7b; no voters:
I20260927 10:48:54.520041 30389 leader_election.cc:288] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:54.520160 30394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:54.520444 30394 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [term 1 LEADER]: Becoming Leader. State: Replica: d0330df7aff847c8ba83b2c98fcb3f7b, State: Running, Role: LEADER
I20260927 10:48:54.520756 30394 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42689 } }
I20260927 10:48:54.521337 30389 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:54.522507 30396 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0330df7aff847c8ba83b2c98fcb3f7b. Latest consensus state: current_term: 1 leader_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42689 } } }
I20260927 10:48:54.522640 30396 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:54.522914 30395 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0330df7aff847c8ba83b2c98fcb3f7b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 42689 } } }
I20260927 10:48:54.523020 30395 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:54.522991 30403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:54.526206 30403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:54.531144 30403 catalog_manager.cc:1347] Generated new cluster ID: e4928ecb69934d6d8b245ac3854f52a2
I20260927 10:48:54.531209 30403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:54.542943 30403 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:54.543848 30403 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:54.551936 30403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b: Generated new TSK 0
I20260927 10:48:54.552677 30403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:54.558897 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.193:0
--local_ip_for_outbound_sockets=127.10.96.193
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42689
--builtin_ntp_servers=127.10.96.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 10:48:54.670974 30413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:54.671231 30413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:54.671270 30413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:54.675321 30413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:54.675371 30413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:54.675426 30413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.193
I20260927 10:48:54.680271 30413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.96.193
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.193
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:54.681495 30413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:54.682883 30413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:54.692551 30419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:54.692575 30421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:54.692554 30418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:54.692816 30413 server_base.cc:1034] running on GCE node
I20260927 10:48:54.693256 30413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:54.693936 30413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:54.695099 30413 hybrid_clock.cc:648] HybridClock initialized: now 1790506134695086 us; error 25 us; skew 500 ppm
I20260927 10:48:54.697000 30413 webserver.cc:460] Webserver started at http://127.10.96.193:36557/ using document root <none> and password file <none>
I20260927 10:48:54.697623 30413 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:54.697687 30413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:54.697923 30413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:54.699913 30413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data/instance:
uuid: "fca564e55fdb45cf8bbc6e639fea267e"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "7e7ac86768b92a938f18c4518a42c7df"
server_key_iv: "ea90caedd85eaecdef8eb6b6d8c79ba9"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.700493 30413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance:
uuid: "fca564e55fdb45cf8bbc6e639fea267e"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "7e7ac86768b92a938f18c4518a42c7df"
server_key_iv: "ea90caedd85eaecdef8eb6b6d8c79ba9"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.704017 30413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 10:48:54.706367 30427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.707541 30413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:54.707672 30413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal
uuid: "fca564e55fdb45cf8bbc6e639fea267e"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "7e7ac86768b92a938f18c4518a42c7df"
server_key_iv: "ea90caedd85eaecdef8eb6b6d8c79ba9"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.707778 30413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:54.725605 30413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:54.726317 30413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:54.726521 30413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:54.727118 30413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:54.728051 30413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:54.728119 30413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.728186 30413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:54.728238 30413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.737040 30413 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.193:39747
I20260927 10:48:54.737057 30540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.193:39747 every 8 connection(s)
I20260927 10:48:54.738072 30413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/data/info.pb
I20260927 10:48:54.745519 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30413
I20260927 10:48:54.745673 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-0/wal/instance
I20260927 10:48:54.746050 10627 external_mini_cluster.cc:1428] Setting key 5450e24d429300b9a532ee7ba068edf5
I20260927 10:48:54.747716 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.739920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:36079 (local address 127.10.96.254:42689)
0927 10:48:54.740067 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.740070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.741143 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 10:48:54.742580 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.742583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.742647 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.742755 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:54.743643 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.744190 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.745382 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.745784 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.745891 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.747198 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:54.747202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:54.747204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:54.747217 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:54.747239 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:54.747248 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:54.747327 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.747489 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.747568 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 10:48:54.748337 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.194:0
--local_ip_for_outbound_sockets=127.10.96.194
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42689
--builtin_ntp_servers=127.10.96.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:48:54.748817 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.740290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42689 (local address 127.10.96.193:36079)
0927 10:48:54.740985 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.741026 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.742359 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.742931 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.742938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.743023 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:54.743501 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.743513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.744318 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.744323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.745244 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.745259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.745976 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.745981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.746194 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.746903 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:54.746907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:54.746986 (+ 79us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:54.747049 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:54.747335 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:54.747342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:54.747346 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.747466 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.747753 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260927 10:48:54.750106 30541 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42689
I20260927 10:48:54.750370 30541 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:54.751015 30541 heartbeater.cc:511] Master 127.10.96.254:42689 requested a full tablet report, sending...
I20260927 10:48:54.752647 30353 ts_manager.cc:194] Registered new tserver with Master: fca564e55fdb45cf8bbc6e639fea267e (127.10.96.193:39747)
I20260927 10:48:54.753860 30353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.193:36079
W20260927 10:48:54.860347 30544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:54.860605 30544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:54.860643 30544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:54.864732 30544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:54.864789 30544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:54.864893 30544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.194
I20260927 10:48:54.870396 30544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.96.194
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.194
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:54.871518 30544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:54.872956 30544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:54.882758 30550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:54.882853 30544 server_base.cc:1034] running on GCE node
W20260927 10:48:54.882965 30549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:54.883231 30552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:54.883678 30544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:54.884310 30544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:54.885540 30544 hybrid_clock.cc:648] HybridClock initialized: now 1790506134885509 us; error 40 us; skew 500 ppm
I20260927 10:48:54.887447 30544 webserver.cc:460] Webserver started at http://127.10.96.194:39051/ using document root <none> and password file <none>
I20260927 10:48:54.887984 30544 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:54.888044 30544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:54.888207 30544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:54.890458 30544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data/instance:
uuid: "513c38f13e77450194f0a5d2c8f4bd00"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "378964343f832206cc93750730ea8ffe"
server_key_iv: "ab88192be264e91ffe0155f722dc2d15"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.890967 30544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance:
uuid: "513c38f13e77450194f0a5d2c8f4bd00"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "378964343f832206cc93750730ea8ffe"
server_key_iv: "ab88192be264e91ffe0155f722dc2d15"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.894580 30544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:48:54.897037 30558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.898314 30544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:48:54.898408 30544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal
uuid: "513c38f13e77450194f0a5d2c8f4bd00"
format_stamp: "Formatted at 2026-09-27 10:48:54 on dist-test-slave-wmg0"
server_key: "378964343f832206cc93750730ea8ffe"
server_key_iv: "ab88192be264e91ffe0155f722dc2d15"
server_key_version: "encryptionkey@0"
I20260927 10:48:54.898527 30544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:54.942028 30544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:54.942721 30544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:54.942875 30544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:54.943405 30544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:54.944283 30544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:54.944329 30544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.944366 30544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:54.944382 30544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:54.953217 30544 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.194:40203
I20260927 10:48:54.953236 30671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.194:40203 every 8 connection(s)
I20260927 10:48:54.954216 30544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/data/info.pb
I20260927 10:48:54.955658 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30544
I20260927 10:48:54.955793 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal/instance
I20260927 10:48:54.956144 10627 external_mini_cluster.cc:1428] Setting key 1da34e1e15a9082ce6b95f2d1ac0a5d4
I20260927 10:48:54.958374 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.96.195:0
--local_ip_for_outbound_sockets=127.10.96.195
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42689
--builtin_ntp_servers=127.10.96.212:46449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 10:48:54.964041 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.956154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.194:45519 (local address 127.10.96.254:42689)
0927 10:48:54.956298 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:54.956303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:54.957654 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 10:48:54.959189 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:54.959195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:54.959259 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:54.959356 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:54.960220 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.960756 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.961973 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.962215 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.962331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.963470 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:54.963474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:54.963476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:54.963490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:54.963556 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:54.963570 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:54.963639 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:54.963866 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 10:48:54.963919 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:48:54.965435 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:54.956625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42689 (local address 127.10.96.194:45519)
0927 10:48:54.957455 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:54.957552 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:54.958958 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:54.959488 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:54.959494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:54.959586 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:54.960077 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.960089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.960885 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.960890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.961851 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:54.961861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:54.962368 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:54.962372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:54.962545 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:54.963194 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:54.963196 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:54.963239 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:54.963302 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:54.963680 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:54.963694 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:54.963703 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 10:48:54.963872 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 10:48:54.964237 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":145,"threads_started":1}
I20260927 10:48:54.966845 30672 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42689
I20260927 10:48:54.967144 30672 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:54.967711 30672 heartbeater.cc:511] Master 127.10.96.254:42689 requested a full tablet report, sending...
I20260927 10:48:54.968833 30353 ts_manager.cc:194] Registered new tserver with Master: 513c38f13e77450194f0a5d2c8f4bd00 (127.10.96.194:40203)
I20260927 10:48:54.969431 30353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.194:45519
W20260927 10:48:55.071960 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:55.072278 30675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:55.072350 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:55.076481 30675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:55.076565 30675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:55.076653 30675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.96.195
I20260927 10:48:55.081609 30675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:46449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.96.195
--webserver_port=0
--tserver_master_addrs=127.10.96.254:42689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.96.195
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:55.082752 30675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:55.084197 30675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:55.093314 30683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:55.093328 30681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:55.093369 30680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:55.093744 30675 server_base.cc:1034] running on GCE node
I20260927 10:48:55.094195 30675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:55.094830 30675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:55.095986 30675 hybrid_clock.cc:648] HybridClock initialized: now 1790506135095953 us; error 51 us; skew 500 ppm
I20260927 10:48:55.097779 30675 webserver.cc:460] Webserver started at http://127.10.96.195:38543/ using document root <none> and password file <none>
I20260927 10:48:55.098340 30675 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:55.098403 30675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:55.098613 30675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:55.100612 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data/instance:
uuid: "4b18369596a24b9a8ac6a97e2d3b75c8"
format_stamp: "Formatted at 2026-09-27 10:48:55 on dist-test-slave-wmg0"
server_key: "af48846bd5215133ae413fbc9275dd9d"
server_key_iv: "5276b42c8651ec2fa3251cfd7821b570"
server_key_version: "encryptionkey@0"
I20260927 10:48:55.101150 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance:
uuid: "4b18369596a24b9a8ac6a97e2d3b75c8"
format_stamp: "Formatted at 2026-09-27 10:48:55 on dist-test-slave-wmg0"
server_key: "af48846bd5215133ae413fbc9275dd9d"
server_key_iv: "5276b42c8651ec2fa3251cfd7821b570"
server_key_version: "encryptionkey@0"
I20260927 10:48:55.104873 30675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:55.107286 30689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:55.108469 30675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:55.108620 30675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal
uuid: "4b18369596a24b9a8ac6a97e2d3b75c8"
format_stamp: "Formatted at 2026-09-27 10:48:55 on dist-test-slave-wmg0"
server_key: "af48846bd5215133ae413fbc9275dd9d"
server_key_iv: "5276b42c8651ec2fa3251cfd7821b570"
server_key_version: "encryptionkey@0"
I20260927 10:48:55.108731 30675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:55.151746 30675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:55.152503 30675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:55.152724 30675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:55.153347 30675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 10:48:55.154332 30675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 10:48:55.154409 30675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:55.154481 30675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 10:48:55.154541 30675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:55.163578 30675 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.195:43451
I20260927 10:48:55.163596 30802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.195:43451 every 8 connection(s)
I20260927 10:48:55.164481 30675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/data/info.pb
I20260927 10:48:55.165243 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30675
I20260927 10:48:55.165333 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal/instance
I20260927 10:48:55.165719 10627 external_mini_cluster.cc:1428] Setting key 8562ae41ff0b7b19846b1596b85ff7b7
I20260927 10:48:55.173687 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.166522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.195:48475 (local address 127.10.96.254:42689)
0927 10:48:55.166668 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.166673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.167971 (+ 1298us) server_negotiation.cc:407] Connection header received
0927 10:48:55.169359 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.169361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.169407 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.169541 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:55.170382 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.170908 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.171984 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.172197 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.172274 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.173246 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:55.173249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:55.173251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:55.173262 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:55.173282 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:55.173290 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:55.173347 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.173523 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.173576 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:48:55.174496 30805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.167062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:42689 (local address 127.10.96.195:48475)
0927 10:48:55.167836 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.167886 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:55.169171 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:55.169689 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:55.169696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:55.169777 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:48:55.170262 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.170273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.171007 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.171009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.171886 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.171896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.172316 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.172318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.172457 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.173018 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 10:48:55.173019 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:48:55.173059 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 10:48:55.173119 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:48:55.173353 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:48:55.173357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:48:55.173360 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 10:48:55.173444 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 10:48:55.173739 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":137,"threads_started":1}
I20260927 10:48:55.175518 30803 heartbeater.cc:347] Connected to a master server at 127.10.96.254:42689
I20260927 10:48:55.175748 30803 heartbeater.cc:464] Registering TS with master...
I20260927 10:48:55.176281 30803 heartbeater.cc:511] Master 127.10.96.254:42689 requested a full tablet report, sending...
I20260927 10:48:55.177253 30353 ts_manager.cc:194] Registered new tserver with Master: 4b18369596a24b9a8ac6a97e2d3b75c8 (127.10.96.195:43451)
I20260927 10:48:55.177832 30353 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.96.195:48475
I20260927 10:48:55.179642 10627 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 10:48:55.184845 30392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.181586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.10.96.254:42689)
0927 10:48:55.181718 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.181722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.181866 (+ 144us) server_negotiation.cc:407] Connection header received
0927 10:48:55.181938 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.181940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.181978 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.182043 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:55.182248 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.182729 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.183475 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.183680 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.183752 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.184430 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:55.184433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:55.184435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:55.184445 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:55.184463 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:55.184470 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:55.184542 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.184625 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.184668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 10:48:55.187367 30353 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 10:48:55.190297 30353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 10:48:55.204689 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.199197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:43451 (local address 127.0.0.1:46196)
0927 10:48:55.200056 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.200068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:55.200164 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:55.200645 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:55.200647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:55.200665 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:55.200751 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.200757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.201852 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.201859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.203383 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.203392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.204142 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.204146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.204246 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.204281 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 10:48:55.204391 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 10:48:55.204516 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":131,"spinlock_wait_cycles":6272,"thread_start_us":91,"threads_started":1}
I20260927 10:48:55.204691 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.199397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.10.96.195:43451)
0927 10:48:55.200206 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.200220 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.200250 (+ 30us) server_negotiation.cc:407] Connection header received
0927 10:48:55.200323 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.200328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.200505 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.200668 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:55.200845 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.201760 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.203511 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.204013 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.204143 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.204342 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.204430 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.204531 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":134,"threads_started":1}
I20260927 10:48:55.204874 30816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.198674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39747 (local address 127.0.0.1:39710)
0927 10:48:55.199581 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.199597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:55.199683 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:55.200307 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:55.200310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:55.200327 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:55.200423 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.200428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.201904 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.201907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.203147 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.203156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.204448 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.204454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.204545 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.204576 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 10:48:55.204648 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 10:48:55.204739 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":131,"thread_start_us":94,"threads_started":1}
I20260927 10:48:55.205226 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.198445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:40203 (local address 127.0.0.1:34758)
0927 10:48:55.199107 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.199168 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:55.199309 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:55.200111 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:55.200117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:55.200155 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:55.200312 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.200319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.201894 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.201897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.203310 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.203321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.204491 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.204494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.204587 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.204614 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 10:48:55.204674 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 10:48:55.205107 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":390,"spinlock_wait_cycles":8448,"thread_start_us":97,"threads_started":1}
I20260927 10:48:55.205487 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.198584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.10.96.194:40203)
0927 10:48:55.198965 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.198981 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.199458 (+ 477us) server_negotiation.cc:407] Connection header received
0927 10:48:55.199559 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.199564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.199779 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.199929 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:55.200462 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.201774 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.203430 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.204209 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.204300 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.204444 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.204748 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.204844 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":160,"threads_started":1}
I20260927 10:48:55.205330 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.199171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.10.96.193:39747)
0927 10:48:55.199835 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.199849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.199874 (+ 25us) server_negotiation.cc:407] Connection header received
0927 10:48:55.199942 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.199947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.200126 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.200306 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:55.200548 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.201763 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.203269 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.204070 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.204193 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.204385 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.205067 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.205163 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":128,"threads_started":1}
I20260927 10:48:55.207962 30475 tablet_service.cc:1467] Processing CreateTablet for tablet dd74a5ed1a5547e3a0638ca614ace7bf (DEFAULT_TABLE table=test-table [id=174ec2c4b0c848d4a0bb1104065370ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:55.207995 30606 tablet_service.cc:1467] Processing CreateTablet for tablet dd74a5ed1a5547e3a0638ca614ace7bf (DEFAULT_TABLE table=test-table [id=174ec2c4b0c848d4a0bb1104065370ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:55.207995 30737 tablet_service.cc:1467] Processing CreateTablet for tablet dd74a5ed1a5547e3a0638ca614ace7bf (DEFAULT_TABLE table=test-table [id=174ec2c4b0c848d4a0bb1104065370ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 10:48:55.208971 30475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd74a5ed1a5547e3a0638ca614ace7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:55.208979 30737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd74a5ed1a5547e3a0638ca614ace7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:55.208979 30606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd74a5ed1a5547e3a0638ca614ace7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:55.215787 30820 tablet_bootstrap.cc:492] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: Bootstrap starting.
I20260927 10:48:55.216835 30821 tablet_bootstrap.cc:492] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Bootstrap starting.
I20260927 10:48:55.218282 30820 tablet_bootstrap.cc:654] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:55.218653 30821 tablet_bootstrap.cc:654] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:55.219162 30820 log.cc:824] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:55.219475 30821 log.cc:824] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:55.220366 30822 tablet_bootstrap.cc:492] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Bootstrap starting.
I20260927 10:48:55.221428 30821 tablet_bootstrap.cc:492] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: No bootstrap required, opened a new log
I20260927 10:48:55.221541 30820 tablet_bootstrap.cc:492] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: No bootstrap required, opened a new log
I20260927 10:48:55.221670 30821 ts_tablet_manager.cc:1397] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 10:48:55.221786 30820 ts_tablet_manager.cc:1397] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:55.222539 30822 tablet_bootstrap.cc:654] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:55.223340 30822 log.cc:824] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:55.225761 30821 raft_consensus.cc:348] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.225960 30822 tablet_bootstrap.cc:492] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: No bootstrap required, opened a new log
I20260927 10:48:55.226046 30821 raft_consensus.cc:374] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:55.226109 30821 raft_consensus.cc:729] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b18369596a24b9a8ac6a97e2d3b75c8, State: Initialized, Role: FOLLOWER
I20260927 10:48:55.226171 30822 ts_tablet_manager.cc:1397] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:55.226114 30820 raft_consensus.cc:348] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.226384 30820 raft_consensus.cc:374] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:55.226440 30820 raft_consensus.cc:729] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca564e55fdb45cf8bbc6e639fea267e, State: Initialized, Role: FOLLOWER
I20260927 10:48:55.226621 30821 consensus_queue.cc:260] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.226850 30820 consensus_queue.cc:260] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.227349 30803 heartbeater.cc:503] Master 127.10.96.254:42689 was elected leader, sending a full tablet report...
I20260927 10:48:55.227699 30821 ts_tablet_manager.cc:1428] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 10:48:55.227769 30820 ts_tablet_manager.cc:1428] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 10:48:55.228091 30541 heartbeater.cc:503] Master 127.10.96.254:42689 was elected leader, sending a full tablet report...
I20260927 10:48:55.229924 30822 raft_consensus.cc:348] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.230211 30822 raft_consensus.cc:374] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:55.230266 30822 raft_consensus.cc:729] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513c38f13e77450194f0a5d2c8f4bd00, State: Initialized, Role: FOLLOWER
I20260927 10:48:55.230703 30822 consensus_queue.cc:260] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.231376 30672 heartbeater.cc:503] Master 127.10.96.254:42689 was elected leader, sending a full tablet report...
I20260927 10:48:55.231719 30822 ts_tablet_manager.cc:1428] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 10:48:55.239210 30542 tablet.cc:2369] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:48:55.415462 30804 tablet.cc:2369] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 10:48:55.455375 30673 tablet.cc:2369] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:48:55.500835 30827 raft_consensus.cc:482] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 10:48:55.501065 30827 raft_consensus.cc:504] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.502408 30827 leader_election.cc:288] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 513c38f13e77450194f0a5d2c8f4bd00 (127.10.96.194:40203), 4b18369596a24b9a8ac6a97e2d3b75c8 (127.10.96.195:43451)
I20260927 10:48:55.507159 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.502673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.195:43451 (local address 127.10.96.193:33861)
0927 10:48:55.503066 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.503082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:55.503258 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:55.503673 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:55.503676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:55.503698 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:55.503858 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.503864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.504893 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.504896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.506051 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.506057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.506705 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.506708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.506804 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.506834 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 10:48:55.506922 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 10:48:55.507003 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":44,"thread_start_us":89,"threads_started":1}
I20260927 10:48:55.507141 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.502858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:33861 (local address 127.10.96.195:43451)
0927 10:48:55.503000 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.503004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.503172 (+ 168us) server_negotiation.cc:407] Connection header received
0927 10:48:55.503421 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.503424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.503474 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.503526 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:55.503984 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.504757 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.506168 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.506577 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.506709 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.506759 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.506939 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.507013 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 10:48:55.507315 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.502993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.96.193:53543 (local address 127.10.96.194:40203)
0927 10:48:55.503125 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.503131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.503327 (+ 196us) server_negotiation.cc:407] Connection header received
0927 10:48:55.503421 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.503424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.503474 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.503556 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 10:48:55.503896 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.504428 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.506004 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.506641 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.506763 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.506810 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.507116 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.507195 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:48:55.507804 30757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd74a5ed1a5547e3a0638ca614ace7bf" candidate_uuid: "fca564e55fdb45cf8bbc6e639fea267e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" is_pre_election: true
I20260927 10:48:55.507872 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.502673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:40203 (local address 127.10.96.193:53543)
0927 10:48:55.503209 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.503222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:48:55.503336 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:48:55.503579 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:48:55.503585 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:48:55.503603 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 10:48:55.503763 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.503770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.504540 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.504544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.505849 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:48:55.505860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.506870 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.506874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:48:55.506982 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.507004 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 10:48:55.507650 (+ 646us) client_negotiation.cc:240] Negotiation successful
0927 10:48:55.507726 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":22,"spinlock_wait_cycles":13056,"thread_start_us":98,"threads_started":1}
I20260927 10:48:55.508147 30757 raft_consensus.cc:2454] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca564e55fdb45cf8bbc6e639fea267e in term 0.
I20260927 10:48:55.508675 30429 leader_election.cc:302] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b18369596a24b9a8ac6a97e2d3b75c8, fca564e55fdb45cf8bbc6e639fea267e; no voters:
I20260927 10:48:55.508929 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd74a5ed1a5547e3a0638ca614ace7bf" candidate_uuid: "fca564e55fdb45cf8bbc6e639fea267e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513c38f13e77450194f0a5d2c8f4bd00" is_pre_election: true
I20260927 10:48:55.508975 30827 raft_consensus.cc:2781] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 10:48:55.509137 30827 raft_consensus.cc:482] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 10:48:55.509196 30827 raft_consensus.cc:3037] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:55.509256 30626 raft_consensus.cc:2454] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fca564e55fdb45cf8bbc6e639fea267e in term 0.
I20260927 10:48:55.510453 30827 raft_consensus.cc:504] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.510872 30827 leader_election.cc:288] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [CANDIDATE]: Term 1 election: Requested vote from peers 513c38f13e77450194f0a5d2c8f4bd00 (127.10.96.194:40203), 4b18369596a24b9a8ac6a97e2d3b75c8 (127.10.96.195:43451)
I20260927 10:48:55.511178 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd74a5ed1a5547e3a0638ca614ace7bf" candidate_uuid: "fca564e55fdb45cf8bbc6e639fea267e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513c38f13e77450194f0a5d2c8f4bd00"
I20260927 10:48:55.511251 30757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd74a5ed1a5547e3a0638ca614ace7bf" candidate_uuid: "fca564e55fdb45cf8bbc6e639fea267e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8"
I20260927 10:48:55.511294 30626 raft_consensus.cc:3037] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:55.511417 30757 raft_consensus.cc:3037] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:55.512578 30626 raft_consensus.cc:2454] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca564e55fdb45cf8bbc6e639fea267e in term 1.
I20260927 10:48:55.512653 30757 raft_consensus.cc:2454] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fca564e55fdb45cf8bbc6e639fea267e in term 1.
I20260927 10:48:55.512976 30431 leader_election.cc:302] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513c38f13e77450194f0a5d2c8f4bd00, fca564e55fdb45cf8bbc6e639fea267e; no voters:
I20260927 10:48:55.513244 30827 raft_consensus.cc:2781] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:55.513513 30827 raft_consensus.cc:686] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [term 1 LEADER]: Becoming Leader. State: Replica: fca564e55fdb45cf8bbc6e639fea267e, State: Running, Role: LEADER
I20260927 10:48:55.513841 30827 consensus_queue.cc:237] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } }
I20260927 10:48:55.516976 30353 catalog_manager.cc:5495] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e reported cstate change: term changed from 0 to 1, leader changed from <none> to fca564e55fdb45cf8bbc6e639fea267e (127.10.96.193). New cstate: current_term: 1 leader_uuid: "fca564e55fdb45cf8bbc6e639fea267e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fca564e55fdb45cf8bbc6e639fea267e" member_type: VOTER last_known_addr { host: "127.10.96.193" port: 39747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 } health_report { overall_health: UNKNOWN } } }
I20260927 10:48:55.574327 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.570733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.10.96.193:39747)
0927 10:48:55.570898 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:55.570903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:55.570917 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:48:55.571001 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:55.571003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:55.571083 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:55.571209 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:48:55.571459 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.571962 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.572845 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:55.573069 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:55.573179 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:55.573510 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:48:55.573938 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:48:55.574093 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:55.574128 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 10:48:55.574179 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 10:48:55.580286 30757 raft_consensus.cc:1261] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 1 FOLLOWER]: Refusing update from remote peer fca564e55fdb45cf8bbc6e639fea267e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:55.580286 30626 raft_consensus.cc:1261] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 1 FOLLOWER]: Refusing update from remote peer fca564e55fdb45cf8bbc6e639fea267e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:48:55.581084 30831 consensus_queue.cc:1035] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [LEADER]: Connected to new peer: Peer: permanent_uuid: "513c38f13e77450194f0a5d2c8f4bd00" member_type: VOTER last_known_addr { host: "127.10.96.194" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:55.581303 30827 consensus_queue.cc:1035] T dd74a5ed1a5547e3a0638ca614ace7bf P fca564e55fdb45cf8bbc6e639fea267e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b18369596a24b9a8ac6a97e2d3b75c8" member_type: VOTER last_known_addr { host: "127.10.96.195" port: 43451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:48:55.588779 30836 mvcc.cc:205] Tried to move back new op lower bound from 7333913131328815104 to 7333913131066822656. Current Snapshot: MvccSnapshot[applied={T|T < 7333913131328815104}]
I20260927 10:48:55.589728 30835 mvcc.cc:205] Tried to move back new op lower bound from 7333913131328815104 to 7333913131066822656. Current Snapshot: MvccSnapshot[applied={T|T < 7333913131328815104}]
I20260927 10:48:55.597703 30353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:55.597885 30353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 10:48:55.600252 30353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0330df7aff847c8ba83b2c98fcb3f7b: Sending DeleteTablet for 3 replicas of tablet dd74a5ed1a5547e3a0638ca614ace7bf
I20260927 10:48:55.601024 30475 tablet_service.cc:1514] Processing DeleteTablet for tablet dd74a5ed1a5547e3a0638ca614ace7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:55 UTC) from {username='slave'} at 127.0.0.1:39710
I20260927 10:48:55.601142 30737 tablet_service.cc:1514] Processing DeleteTablet for tablet dd74a5ed1a5547e3a0638ca614ace7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:55 UTC) from {username='slave'} at 127.0.0.1:46196
I20260927 10:48:55.601622 30606 tablet_service.cc:1514] Processing DeleteTablet for tablet dd74a5ed1a5547e3a0638ca614ace7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 10:48:55 UTC) from {username='slave'} at 127.0.0.1:34758
I20260927 10:48:55.601876 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30413
I20260927 10:48:55.602681 30844 tablet_replica.cc:330] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: stopping tablet replica
I20260927 10:48:55.603036 30844 raft_consensus.cc:2229] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:55.603374 30844 raft_consensus.cc:2258] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:55.604322 30845 tablet_replica.cc:330] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: stopping tablet replica
I20260927 10:48:55.604653 30845 raft_consensus.cc:2229] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 10:48:55.604974 30845 raft_consensus.cc:2258] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 10:48:55.605121 30844 ts_tablet_manager.cc:1905] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:55.607012 30845 ts_tablet_manager.cc:1905] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 10:48:55.607452 30844 ts_tablet_manager.cc:1918] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:55.607569 30844 log.cc:1196] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-2/wal/wals/dd74a5ed1a5547e3a0638ca614ace7bf
I20260927 10:48:55.607842 30844 ts_tablet_manager.cc:1939] T dd74a5ed1a5547e3a0638ca614ace7bf P 4b18369596a24b9a8ac6a97e2d3b75c8: Deleting consensus metadata
I20260927 10:48:55.608959 30338 catalog_manager.cc:4841] TS 4b18369596a24b9a8ac6a97e2d3b75c8 (127.10.96.195:43451): tablet dd74a5ed1a5547e3a0638ca614ace7bf (table test-table [id=174ec2c4b0c848d4a0bb1104065370ff]) successfully deleted
W20260927 10:48:55.609730 30337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.96.193:39747 (error 108)
I20260927 10:48:55.610517 30845 ts_tablet_manager.cc:1918] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 10:48:55.610612 30845 log.cc:1196] T dd74a5ed1a5547e3a0638ca614ace7bf P 513c38f13e77450194f0a5d2c8f4bd00: Deleting WAL directory at /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790505884435945-10627-0/minicluster-data/ts-1/wal/wals/dd74a5ed1a5547e3a0638ca614ace7bf
I20260927 10:48:55.610603 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30544
I20260927 10:48:55.610749 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.610429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.193:39747 (local address 127.0.0.1:39728)
0927 10:48:55.610572 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.610651 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:48:55.611310 30337 catalog_manager.cc:4584] TS fca564e55fdb45cf8bbc6e639fea267e (127.10.96.193:39747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd74a5ed1a5547e3a0638ca614ace7bf: Network error: Client connection negotiation failed: client connection to 127.10.96.193:39747: connect: Connection refused (error 111)
I20260927 10:48:55.616552 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30675
I20260927 10:48:55.617574 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:55.616698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.194:40203 (local address 127.0.0.1:34772)
0927 10:48:55.617391 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 10:48:55.617476 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.194:40203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595}
I20260927 10:48:55.622200 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30322
2026-09-27T10:48:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1321 ms)
[----------] 6 tests from EncryptionPolicyTest (10628 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:55 dist-test-slave-wmg0 krb5kdc[30849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:55 dist-test-slave-wmg0 krb5kdc[30849](info): set up 2 sockets
Sep 27 10:48:55 dist-test-slave-wmg0 krb5kdc[30849](info): commencing operation
krb5kdc: starting...
W20260927 10:48:56.657608 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:48:56 dist-test-slave-wmg0 krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:56.777153 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:56.888609 30862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:56.888885 30862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:56.888928 30862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:56.892863 30862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:56.892925 30862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:56.892944 30862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:56.892964 30862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:56.892979 30862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:56.898204 30862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:56.899197 30862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:56.900174 30862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:56.908483 30868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:56.908497 30867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:56.908722 30862 server_base.cc:1034] running on GCE node
W20260927 10:48:56.908497 30870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:56.909359 30862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:56.910458 30862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:56.911628 30862 hybrid_clock.cc:648] HybridClock initialized: now 1790506136911594 us; error 45 us; skew 500 ppm
Sep 27 10:48:56 dist-test-slave-wmg0 krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:56.920372 30862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:56.921603 30862 webserver.cc:460] Webserver started at http://127.0.0.1:33055/ using document root <none> and password file <none>
I20260927 10:48:56.922202 30862 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:56.922292 30862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:56.922529 30862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:56.924621 30862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "7b059ebb61fd467ea80a5445b012f107"
format_stamp: "Formatted at 2026-09-27 10:48:56 on dist-test-slave-wmg0"
server_key: "809b8320d8d965fba8fb058a7cc135c1"
server_key_iv: "a556bb7bcccc3c549961547f383d0fef"
server_key_version: "encryptionkey@0"
I20260927 10:48:56.925196 30862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "7b059ebb61fd467ea80a5445b012f107"
format_stamp: "Formatted at 2026-09-27 10:48:56 on dist-test-slave-wmg0"
server_key: "809b8320d8d965fba8fb058a7cc135c1"
server_key_iv: "a556bb7bcccc3c549961547f383d0fef"
server_key_version: "encryptionkey@0"
I20260927 10:48:56.928897 30862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:48:56.931490 30877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:56.932730 30862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:56.932875 30862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "7b059ebb61fd467ea80a5445b012f107"
format_stamp: "Formatted at 2026-09-27 10:48:56 on dist-test-slave-wmg0"
server_key: "809b8320d8d965fba8fb058a7cc135c1"
server_key_iv: "a556bb7bcccc3c549961547f383d0fef"
server_key_version: "encryptionkey@0"
I20260927 10:48:56.932996 30862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:56.957777 30862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:56.960984 30862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:56.961199 30862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:56.968412 30862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46271
I20260927 10:48:56.968436 30929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46271 every 8 connection(s)
I20260927 10:48:56.969353 30862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:56.972105 30930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:56.973590 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30862
I20260927 10:48:56.973744 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:56.974076 10627 external_mini_cluster.cc:1428] Setting key aab1a90af2f34fd182d12fa056eb1feb
I20260927 10:48:56.978317 30930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107: Bootstrap starting.
I20260927 10:48:56.980520 30930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:56.981356 30930 log.cc:824] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:48:56 dist-test-slave-wmg0 krb5kdc[30849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:48:56.983376 30930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107: No bootstrap required, opened a new log
I20260927 10:48:56.987030 30930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b059ebb61fd467ea80a5445b012f107" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46271 } }
I20260927 10:48:56.987282 30930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:56.987340 30930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b059ebb61fd467ea80a5445b012f107, State: Initialized, Role: FOLLOWER
I20260927 10:48:56.987783 30930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b059ebb61fd467ea80a5445b012f107" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46271 } }
I20260927 10:48:56.987907 30930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:56.987989 30930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:56.988102 30930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:56.988294 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:56.975500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46795 (local address 127.0.0.1:46271)
0927 10:48:56.975959 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:56.975974 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:56.976009 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:48:56.976642 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:56.976658 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:56.977003 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:56.977338 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:56.978185 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:56.979183 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:56.980037 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:56.980509 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:56.980678 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:56.983481 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:56.983491 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:56.983503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:56.983529 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:56.985203 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:56.985787 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:56.985791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:56.985796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:56.985905 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:56.986247 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:56.986250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:56.986256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:56.986634 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:56.986867 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:56.987175 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 10:48:56.987440 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260927 10:48:56.989024 30930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b059ebb61fd467ea80a5445b012f107" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46271 } }
I20260927 10:48:56.989405 30930 leader_election.cc:302] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b059ebb61fd467ea80a5445b012f107; no voters:
I20260927 10:48:56.989784 30930 leader_election.cc:288] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:56.989964 30935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:56.990715 30935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [term 1 LEADER]: Becoming Leader. State: Replica: 7b059ebb61fd467ea80a5445b012f107, State: Running, Role: LEADER
I20260927 10:48:56.991011 30935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b059ebb61fd467ea80a5445b012f107" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46271 } }
I20260927 10:48:56.991036 30930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:56.991391 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:56.994660 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b059ebb61fd467ea80a5445b012f107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b059ebb61fd467ea80a5445b012f107" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46271 } } }
I20260927 10:48:56.994781 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:56.994962 30937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b059ebb61fd467ea80a5445b012f107. Latest consensus state: current_term: 1 leader_uuid: "7b059ebb61fd467ea80a5445b012f107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b059ebb61fd467ea80a5445b012f107" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46271 } } }
I20260927 10:48:56.995244 30955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:56.995780 30937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:56.998852 30955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:57.002106 30933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:56.993389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50065 (local address 127.0.0.1:46271)
0927 10:48:56.993531 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:56.993535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:56.993656 (+ 121us) server_negotiation.cc:407] Connection header received
0927 10:48:56.993798 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:56.993801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:56.993848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:56.993926 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:56.994834 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:56.995509 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:56.996877 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:56.997135 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:56.997258 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:56.998730 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:56.998734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:56.998736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:48:56.998760 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:57.000299 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:57.000781 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:57.000784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:57.000786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:57.000873 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:48:57.001229 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:48:57.001232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:48:57.001234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:48:57.001407 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:57.001603 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:57.001850 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 10:48:57.001952 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:48:57.004849 30955 catalog_manager.cc:1347] Generated new cluster ID: cbcab92303d84c78ac932631fc5a7d50
I20260927 10:48:57.004910 30955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:57.027611 30955 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:57.028883 30955 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:57.040254 30955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b059ebb61fd467ea80a5445b012f107: Generated new TSK 0
I20260927 10:48:57.041005 30955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:57.049345 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30862
W20260927 10:48:57.055001 30847 connection.cc:535] client connection to 127.0.0.1:46271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46271): Connection reset by peer (error 104)
2026-09-27T10:48:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:57 dist-test-slave-wmg0 krb5kdc[30961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:57 dist-test-slave-wmg0 krb5kdc[30961](info): set up 2 sockets
Sep 27 10:48:57 dist-test-slave-wmg0 krb5kdc[30961](info): commencing operation
krb5kdc: starting...
W20260927 10:48:58.081775 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:48:58 dist-test-slave-wmg0 krb5kdc[30961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:58.204414 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:58.316303 30974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:58.316668 30974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:58.316749 30974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:58.321161 30974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:58.321270 30974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:58.321311 30974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:58.321373 30974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:58.321411 30974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:58.327108 30974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:58.328325 30974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:58.329406 30974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:58.338160 30979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:58.338158 30982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:58.338239 30974 server_base.cc:1034] running on GCE node
W20260927 10:48:58.338127 30980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:58.338950 30974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:58.340160 30974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:58.341389 30974 hybrid_clock.cc:648] HybridClock initialized: now 1790506138341362 us; error 55 us; skew 500 ppm
Sep 27 10:48:58 dist-test-slave-wmg0 krb5kdc[30961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:58.344424 30974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:58.345408 30974 webserver.cc:460] Webserver started at http://127.0.0.1:39429/ using document root <none> and password file <none>
I20260927 10:48:58.346089 30974 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:58.346184 30974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:58.346417 30974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:58.348489 30974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "d5153c8e80cd44609391e2005bcb6610"
format_stamp: "Formatted at 2026-09-27 10:48:58 on dist-test-slave-wmg0"
server_key: "cb992a8ebb18c94d904bac3c5583bf52"
server_key_iv: "c4b280857de5d8585c3c60fbb5bc3024"
server_key_version: "encryptionkey@0"
I20260927 10:48:58.349061 30974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "d5153c8e80cd44609391e2005bcb6610"
format_stamp: "Formatted at 2026-09-27 10:48:58 on dist-test-slave-wmg0"
server_key: "cb992a8ebb18c94d904bac3c5583bf52"
server_key_iv: "c4b280857de5d8585c3c60fbb5bc3024"
server_key_version: "encryptionkey@0"
I20260927 10:48:58.352874 30974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 10:48:58.355388 30989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:58.356719 30974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:48:58.356884 30974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "d5153c8e80cd44609391e2005bcb6610"
format_stamp: "Formatted at 2026-09-27 10:48:58 on dist-test-slave-wmg0"
server_key: "cb992a8ebb18c94d904bac3c5583bf52"
server_key_iv: "c4b280857de5d8585c3c60fbb5bc3024"
server_key_version: "encryptionkey@0"
I20260927 10:48:58.357009 30974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:58.387751 30974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:58.391502 30974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:58.391726 30974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:58.399382 30974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44725
I20260927 10:48:58.399403 31041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44725 every 8 connection(s)
I20260927 10:48:58.400460 30974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:58.401361 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 30974
I20260927 10:48:58.401554 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:58.401890 10627 external_mini_cluster.cc:1428] Setting key e1b300a49132e367ba6186167fa99578
I20260927 10:48:58.404057 31042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:58.409541 31042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610: Bootstrap starting.
I20260927 10:48:58.411224 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:58.403440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45331 (local address 127.0.0.1:44725)
0927 10:48:58.403922 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:58.403937 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:58.403973 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:48:58.404741 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:58.404770 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:58.405148 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:58.405519 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:58.405924 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:58.406897 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:58.407827 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:58.408307 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:58.408486 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:58.409444 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:58.409449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:58.409521 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:58.409961 (+ 440us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:58.410022 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:58.410045 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:58.410186 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:58.410363 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 10:48:58.410518 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":145,"threads_started":1}
I20260927 10:48:58.411999 31042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:58.412727 31042 log.cc:824] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:58.414295 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:58.414845 31042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610: No bootstrap required, opened a new log
I20260927 10:48:58.419329 31042 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5153c8e80cd44609391e2005bcb6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44725 } }
I20260927 10:48:58.419678 31042 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:58.419775 31042 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5153c8e80cd44609391e2005bcb6610, State: Initialized, Role: FOLLOWER
I20260927 10:48:58.420291 31042 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5153c8e80cd44609391e2005bcb6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44725 } }
I20260927 10:48:58.420464 31042 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:58.420569 31042 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:58.420688 31042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:58.420943 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:58.416454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38049 (local address 127.0.0.1:44725)
0927 10:48:58.416580 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:58.416583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:58.416796 (+ 213us) server_negotiation.cc:407] Connection header received
0927 10:48:58.416893 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:58.416897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:58.416951 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:58.417059 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:58.417275 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:58.418103 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:58.418932 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:48:58.419256 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:48:58.419427 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:48:58.420383 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:58.420387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:58.420390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:58.420402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:58.420431 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:58.420444 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:58.420542 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:58.420775 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 10:48:58.420819 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 10:48:58.421917 31042 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5153c8e80cd44609391e2005bcb6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44725 } }
I20260927 10:48:58.422271 31042 leader_election.cc:302] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5153c8e80cd44609391e2005bcb6610; no voters:
I20260927 10:48:58.422592 31042 leader_election.cc:288] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:58.422751 31052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:58.423005 31052 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [term 1 LEADER]: Becoming Leader. State: Replica: d5153c8e80cd44609391e2005bcb6610, State: Running, Role: LEADER
I20260927 10:48:58.423389 31052 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5153c8e80cd44609391e2005bcb6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44725 } }
I20260927 10:48:58.423897 31042 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:58.425050 31053 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5153c8e80cd44609391e2005bcb6610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5153c8e80cd44609391e2005bcb6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44725 } } }
I20260927 10:48:58.425174 31053 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:58.425462 31054 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5153c8e80cd44609391e2005bcb6610. Latest consensus state: current_term: 1 leader_uuid: "d5153c8e80cd44609391e2005bcb6610" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5153c8e80cd44609391e2005bcb6610" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44725 } } }
I20260927 10:48:58.425570 31054 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610 [sys.catalog]: This master's current role is: LEADER
W20260927 10:48:58.428717 31067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:48:58.429075 31067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:48:58.429157 31061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:58.429931 31061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:58.435020 31061 catalog_manager.cc:1347] Generated new cluster ID: 6ef2bb99c99c41d58bcea7f794d74489
I20260927 10:48:58.435083 31061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:58.457942 31061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:58.458977 31061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:58.464653 31061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5153c8e80cd44609391e2005bcb6610: Generated new TSK 0
I20260927 10:48:58.465561 31061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:58.477218 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 30974
W20260927 10:48:58.483080 30959 connection.cc:535] client connection to 127.0.0.1:44725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44725): Connection reset by peer (error 104)
2026-09-27T10:48:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:58 dist-test-slave-wmg0 krb5kdc[31073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:58 dist-test-slave-wmg0 krb5kdc[31073](info): set up 2 sockets
Sep 27 10:48:58 dist-test-slave-wmg0 krb5kdc[31073](info): commencing operation
krb5kdc: starting...
W20260927 10:48:59.513849 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:48:59 dist-test-slave-wmg0 krb5kdc[31073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:48:59.632498 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:48:59.742386 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:48:59.742676 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:48:59.742751 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:48:59.746790 31086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:48:59.746853 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:48:59.746873 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:48:59.746891 31086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:48:59.746942 31086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:48:59.752213 31086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:48:59.753262 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:48:59.754319 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:48:59.762913 31094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:48:59.762909 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:59.762959 31086 server_base.cc:1034] running on GCE node
W20260927 10:48:59.762909 31091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:48:59.763723 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:48:59.764810 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:48:59.765998 31086 hybrid_clock.cc:648] HybridClock initialized: now 1790506139765963 us; error 52 us; skew 500 ppm
Sep 27 10:48:59 dist-test-slave-wmg0 krb5kdc[31073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:48:59.768644 31086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:48:59.769611 31086 webserver.cc:460] Webserver started at http://127.0.0.1:38687/ using document root <none> and password file <none>
I20260927 10:48:59.770169 31086 fs_manager.cc:359] Metadata directory not provided
I20260927 10:48:59.770256 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:48:59.770473 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:48:59.772497 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "b18ed60ce24c45c5ba6834c06f619ae6"
format_stamp: "Formatted at 2026-09-27 10:48:59 on dist-test-slave-wmg0"
server_key: "7c6bc8c61362332df1fc6722e4817da8"
server_key_iv: "534c085ad9466f1b9e28f4f6b09cc097"
server_key_version: "encryptionkey@0"
I20260927 10:48:59.773067 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "b18ed60ce24c45c5ba6834c06f619ae6"
format_stamp: "Formatted at 2026-09-27 10:48:59 on dist-test-slave-wmg0"
server_key: "7c6bc8c61362332df1fc6722e4817da8"
server_key_iv: "534c085ad9466f1b9e28f4f6b09cc097"
server_key_version: "encryptionkey@0"
I20260927 10:48:59.776871 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 10:48:59.779381 31101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:48:59.780828 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 10:48:59.780988 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "b18ed60ce24c45c5ba6834c06f619ae6"
format_stamp: "Formatted at 2026-09-27 10:48:59 on dist-test-slave-wmg0"
server_key: "7c6bc8c61362332df1fc6722e4817da8"
server_key_iv: "534c085ad9466f1b9e28f4f6b09cc097"
server_key_version: "encryptionkey@0"
I20260927 10:48:59.781114 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:48:59.785795 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:48:59.788698 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:48:59.788888 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:48:59.796159 31086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45083
I20260927 10:48:59.796185 31153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45083 every 8 connection(s)
I20260927 10:48:59.797583 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:48:59.798372 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31086
I20260927 10:48:59.798516 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:48:59.798808 10627 external_mini_cluster.cc:1428] Setting key 5641e2ec39481907dbd64d08ceab5782
I20260927 10:48:59.801038 31154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:48:59.804563 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:59.800228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41163 (local address 127.0.0.1:45083)
0927 10:48:59.800640 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:59.800659 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:59.800707 (+ 48us) server_negotiation.cc:407] Connection header received
0927 10:48:59.801594 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:59.801618 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:59.801993 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:59.802361 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:59.803155 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:59.803161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:59.803176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:59.803188 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:59.803256 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:59.803281 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:59.803424 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:59.803492 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:48:59.803642 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260927 10:48:59.807485 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:48:59.807932 31154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6: Bootstrap starting.
I20260927 10:48:59.810307 31154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6: Neither blocks nor log segments found. Creating new log.
I20260927 10:48:59.811033 31154 log.cc:824] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6: Log is configured to *not* fsync() on all Append() calls
I20260927 10:48:59.813170 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:48:59.809563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57763 (local address 127.0.0.1:45083)
0927 10:48:59.809671 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:48:59.809674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:48:59.811614 (+ 1940us) server_negotiation.cc:407] Connection header received
0927 10:48:59.811789 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:48:59.811791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:48:59.811821 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:48:59.811887 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:48:59.812656 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:48:59.812659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:48:59.812661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:48:59.812671 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:48:59.812696 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:48:59.812706 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:48:59.812785 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 10:48:59.812948 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 10:48:59.813023 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:48:59.813280 31154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6: No bootstrap required, opened a new log
I20260927 10:48:59.816824 31154 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260927 10:48:59.817058 31154 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:48:59.817127 31154 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18ed60ce24c45c5ba6834c06f619ae6, State: Initialized, Role: FOLLOWER
I20260927 10:48:59.817589 31154 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260927 10:48:59.817740 31154 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:48:59.817811 31154 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:48:59.817926 31154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:48:59.818846 31154 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260927 10:48:59.819178 31154 leader_election.cc:302] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18ed60ce24c45c5ba6834c06f619ae6; no voters:
I20260927 10:48:59.819484 31154 leader_election.cc:288] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:48:59.819607 31164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:48:59.819825 31164 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [term 1 LEADER]: Becoming Leader. State: Replica: b18ed60ce24c45c5ba6834c06f619ae6, State: Running, Role: LEADER
I20260927 10:48:59.820161 31164 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260927 10:48:59.820765 31154 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:48:59.821926 31166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18ed60ce24c45c5ba6834c06f619ae6. Latest consensus state: current_term: 1 leader_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } } }
I20260927 10:48:59.821966 31165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18ed60ce24c45c5ba6834c06f619ae6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } } }
I20260927 10:48:59.822067 31166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:59.822067 31165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6 [sys.catalog]: This master's current role is: LEADER
I20260927 10:48:59.822407 31173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:48:59.825666 31173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:48:59.831108 31173 catalog_manager.cc:1347] Generated new cluster ID: a08feb4c2fe14672acfe8762204bead5
I20260927 10:48:59.831182 31173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:48:59.837682 31173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:48:59.838065 31173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:48:59.848646 31173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18ed60ce24c45c5ba6834c06f619ae6: Generated new TSK 0
I20260927 10:48:59.849500 31173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:48:59.854944 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31086
2026-09-27T10:48:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:48:59 dist-test-slave-wmg0 krb5kdc[31185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:48:59 dist-test-slave-wmg0 krb5kdc[31185](info): set up 2 sockets
Sep 27 10:48:59 dist-test-slave-wmg0 krb5kdc[31185](info): commencing operation
krb5kdc: starting...
W20260927 10:49:00.893697 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:00 dist-test-slave-wmg0 krb5kdc[31185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:01.015915 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:01.127288 31198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:01.127565 31198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:01.127650 31198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:01.131896 31198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:01.131964 31198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:01.131989 31198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:01.132005 31198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:01.132045 31198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:01.137488 31198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:01.138569 31198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:01.139611 31198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:01.148521 31198 server_base.cc:1034] running on GCE node
W20260927 10:49:01.148463 31204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:01.148473 31203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:01.148460 31206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:01.149299 31198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:01.150499 31198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:01.151685 31198 hybrid_clock.cc:648] HybridClock initialized: now 1790506141151651 us; error 54 us; skew 500 ppm
Sep 27 10:49:01 dist-test-slave-wmg0 krb5kdc[31185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:01.154659 31198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:01.155617 31198 webserver.cc:460] Webserver started at http://127.0.0.1:36515/ using document root <none> and password file <none>
I20260927 10:49:01.156217 31198 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:01.156311 31198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:01.156544 31198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:01.158685 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "d74e312703194a4eb69beb3dfb580076"
format_stamp: "Formatted at 2026-09-27 10:49:01 on dist-test-slave-wmg0"
server_key: "59801b0b9b3dd33dafcc63ca6a2b351f"
server_key_iv: "dcd808471871a8dfa3193c05f1cd46ba"
server_key_version: "encryptionkey@0"
I20260927 10:49:01.159256 31198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "d74e312703194a4eb69beb3dfb580076"
format_stamp: "Formatted at 2026-09-27 10:49:01 on dist-test-slave-wmg0"
server_key: "59801b0b9b3dd33dafcc63ca6a2b351f"
server_key_iv: "dcd808471871a8dfa3193c05f1cd46ba"
server_key_version: "encryptionkey@0"
I20260927 10:49:01.163084 31198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:49:01.165602 31213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:01.166854 31198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:01.167044 31198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "d74e312703194a4eb69beb3dfb580076"
format_stamp: "Formatted at 2026-09-27 10:49:01 on dist-test-slave-wmg0"
server_key: "59801b0b9b3dd33dafcc63ca6a2b351f"
server_key_iv: "dcd808471871a8dfa3193c05f1cd46ba"
server_key_version: "encryptionkey@0"
I20260927 10:49:01.167161 31198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:01.181855 31198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:01.185156 31198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:01.185374 31198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:01.192634 31198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44539
I20260927 10:49:01.192647 31265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44539 every 8 connection(s)
I20260927 10:49:01.193641 31198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:01.196455 31266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:01.202046 31266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076: Bootstrap starting.
I20260927 10:49:01.202462 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31198
I20260927 10:49:01.202629 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:01.202910 10627 external_mini_cluster.cc:1428] Setting key 73aa3121b117f91785e649e040011f35
I20260927 10:49:01.204797 31266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:01.205600 31266 log.cc:824] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:01.207456 31266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076: No bootstrap required, opened a new log
I20260927 10:49:01.211035 31266 raft_consensus.cc:348] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e312703194a4eb69beb3dfb580076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260927 10:49:01.211321 31266 raft_consensus.cc:374] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:01.211390 31266 raft_consensus.cc:729] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74e312703194a4eb69beb3dfb580076, State: Initialized, Role: FOLLOWER
Sep 27 10:49:01 dist-test-slave-wmg0 krb5kdc[31185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:49:01.211863 31266 consensus_queue.cc:260] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e312703194a4eb69beb3dfb580076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260927 10:49:01.212040 31266 raft_consensus.cc:388] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:01.212127 31266 raft_consensus.cc:482] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:01.212246 31266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:01.213492 31266 raft_consensus.cc:504] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e312703194a4eb69beb3dfb580076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260927 10:49:01.213882 31266 leader_election.cc:302] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d74e312703194a4eb69beb3dfb580076; no voters:
I20260927 10:49:01.214212 31266 leader_election.cc:288] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:01.214308 31271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:01.214561 31271 raft_consensus.cc:686] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [term 1 LEADER]: Becoming Leader. State: Replica: d74e312703194a4eb69beb3dfb580076, State: Running, Role: LEADER
I20260927 10:49:01.214886 31271 consensus_queue.cc:237] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e312703194a4eb69beb3dfb580076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } }
I20260927 10:49:01.215557 31266 sys_catalog.cc:564] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:01.216653 31272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d74e312703194a4eb69beb3dfb580076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e312703194a4eb69beb3dfb580076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } } }
I20260927 10:49:01.216650 31273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d74e312703194a4eb69beb3dfb580076. Latest consensus state: current_term: 1 leader_uuid: "d74e312703194a4eb69beb3dfb580076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74e312703194a4eb69beb3dfb580076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44539 } } }
I20260927 10:49:01.216799 31272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:01.216820 31273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:01.217201 31280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:01.218681 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:01.204284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41983 (local address 127.0.0.1:44539)
0927 10:49:01.204707 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:01.204722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:01.204756 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:49:01.205444 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:01.205513 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:01.205897 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:01.206228 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:01.207036 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:01.208068 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:01.208937 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:01.209412 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:01.209640 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:01.212462 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:01.212471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:01.212489 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:01.212521 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:01.214635 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:01.215185 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:01.215190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:01.215196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:01.215334 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:01.215707 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:01.215710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:01.215712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:01.216285 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:01.217240 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:01.217402 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:49:01.217735 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":124,"threads_started":1}
I20260927 10:49:01.220249 31280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:01.221617 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:01.225390 31280 catalog_manager.cc:1347] Generated new cluster ID: acd8e49d1ddd44758bbaad5c23f6b6ee
I20260927 10:49:01.225543 31280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:01.232965 31269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:01.223854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56247 (local address 127.0.0.1:44539)
0927 10:49:01.223965 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:01.223970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:01.224367 (+ 397us) server_negotiation.cc:407] Connection header received
0927 10:49:01.224408 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:01.224412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:01.224458 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:01.224529 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:01.225484 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:01.226291 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:01.227377 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:01.227733 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:01.227843 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:01.229368 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:01.229372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:01.229375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:01.229402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:01.231135 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:01.231593 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:01.231597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:01.231599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:01.231691 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:01.231958 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:01.231961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:01.231964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:01.232187 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:01.232333 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:01.232532 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 10:49:01.232796 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 10:49:01.254666 31280 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:01.255806 31280 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:01.259680 31280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d74e312703194a4eb69beb3dfb580076: Generated new TSK 0
I20260927 10:49:01.260425 31280 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:01.268147 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31198
2026-09-27T10:49:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:01 dist-test-slave-wmg0 krb5kdc[31297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:01 dist-test-slave-wmg0 krb5kdc[31297](info): set up 2 sockets
Sep 27 10:49:01 dist-test-slave-wmg0 krb5kdc[31297](info): commencing operation
krb5kdc: starting...
W20260927 10:49:02.309813 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:02 dist-test-slave-wmg0 krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:02.432381 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:02.541684 31310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:02.541989 31310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:02.542064 31310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:02.546063 31310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:02.546142 31310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:02.546178 31310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:02.546236 31310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:02.546275 31310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:02.551455 31310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:02.552501 31310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:02.553583 31310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:02.561852 31316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:02.561897 31310 server_base.cc:1034] running on GCE node
W20260927 10:49:02.561846 31318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:02.561851 31315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:02.562670 31310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:02.563660 31310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:02.564819 31310 hybrid_clock.cc:648] HybridClock initialized: now 1790506142564801 us; error 30 us; skew 500 ppm
Sep 27 10:49:02 dist-test-slave-wmg0 krb5kdc[31297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:02.567477 31310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:02.568288 31310 webserver.cc:460] Webserver started at http://127.0.0.1:34157/ using document root <none> and password file <none>
I20260927 10:49:02.568771 31310 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:02.568827 31310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:02.569051 31310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:02.571327 31310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "9521ff26797643f484296ad25b67fa75"
format_stamp: "Formatted at 2026-09-27 10:49:02 on dist-test-slave-wmg0"
server_key: "7c5e8cb0b40e789c4c994290fc47483c"
server_key_iv: "ded486671cd5cab180429f641f42eb9f"
server_key_version: "encryptionkey@0"
I20260927 10:49:02.571888 31310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "9521ff26797643f484296ad25b67fa75"
format_stamp: "Formatted at 2026-09-27 10:49:02 on dist-test-slave-wmg0"
server_key: "7c5e8cb0b40e789c4c994290fc47483c"
server_key_iv: "ded486671cd5cab180429f641f42eb9f"
server_key_version: "encryptionkey@0"
I20260927 10:49:02.575465 31310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:49:02.577757 31325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:02.579034 31310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 10:49:02.579185 31310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "9521ff26797643f484296ad25b67fa75"
format_stamp: "Formatted at 2026-09-27 10:49:02 on dist-test-slave-wmg0"
server_key: "7c5e8cb0b40e789c4c994290fc47483c"
server_key_iv: "ded486671cd5cab180429f641f42eb9f"
server_key_version: "encryptionkey@0"
I20260927 10:49:02.579300 31310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:02.613335 31310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:02.616740 31310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:02.616971 31310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:02.624291 31310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40261
I20260927 10:49:02.624325 31377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40261 every 8 connection(s)
I20260927 10:49:02.625303 31310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:02.628155 31378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:02.628974 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31310
I20260927 10:49:02.629119 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:02.629429 10627 external_mini_cluster.cc:1428] Setting key 5674a69a9e2452b666b368bad66d6216
I20260927 10:49:02.634707 31378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75: Bootstrap starting.
I20260927 10:49:02.637331 31378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:02.638159 31378 log.cc:824] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:02.638629 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:02.630824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59295 (local address 127.0.0.1:40261)
0927 10:49:02.631281 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:02.631295 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:02.631330 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:02.631978 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:02.631996 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:02.632346 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:02.632627 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:02.632980 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:02.634019 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:02.635179 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:02.635642 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:02.635771 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:02.636742 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:02.636746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:02.636760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:02.637179 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:02.637233 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:02.637255 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:02.637332 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:02.637591 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 10:49:02.637831 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":145,"threads_started":1}
I20260927 10:49:02.640182 31378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75: No bootstrap required, opened a new log
I20260927 10:49:02.641528 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:02.644945 31378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9521ff26797643f484296ad25b67fa75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40261 } }
I20260927 10:49:02.645236 31378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:02.645283 31378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9521ff26797643f484296ad25b67fa75, State: Initialized, Role: FOLLOWER
I20260927 10:49:02.645771 31378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9521ff26797643f484296ad25b67fa75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40261 } }
I20260927 10:49:02.645910 31378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:02.645995 31378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:02.646098 31378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:02.647305 31378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9521ff26797643f484296ad25b67fa75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40261 } }
I20260927 10:49:02.647657 31378 leader_election.cc:302] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9521ff26797643f484296ad25b67fa75; no voters:
I20260927 10:49:02.648001 31378 leader_election.cc:288] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:02.648075 31388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:02.648272 31388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [term 1 LEADER]: Becoming Leader. State: Replica: 9521ff26797643f484296ad25b67fa75, State: Running, Role: LEADER
I20260927 10:49:02.648411 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:02.643881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35539 (local address 127.0.0.1:40261)
0927 10:49:02.644023 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:02.644028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:02.644218 (+ 190us) server_negotiation.cc:407] Connection header received
0927 10:49:02.644317 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:02.644320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:02.644373 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:02.644451 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:02.644757 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:02.645551 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:02.646552 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:02.646905 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:02.647017 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:02.647886 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:02.647889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:02.647892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:02.647904 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:02.647932 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:02.647946 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:02.648031 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:02.648159 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 10:49:02.648221 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:49:02.648650 31388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9521ff26797643f484296ad25b67fa75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40261 } }
I20260927 10:49:02.649405 31378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:02.650589 31390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9521ff26797643f484296ad25b67fa75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9521ff26797643f484296ad25b67fa75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40261 } } }
I20260927 10:49:02.650740 31390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:02.651124 31397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:02.650591 31389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9521ff26797643f484296ad25b67fa75. Latest consensus state: current_term: 1 leader_uuid: "9521ff26797643f484296ad25b67fa75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9521ff26797643f484296ad25b67fa75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40261 } } }
I20260927 10:49:02.651438 31389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:02.654560 31397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:02.660128 31397 catalog_manager.cc:1347] Generated new cluster ID: eb6280383e364483ae4ad739dd644ffc
I20260927 10:49:02.660194 31397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:02.674319 31397 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:02.675258 31397 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:02.680301 31397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9521ff26797643f484296ad25b67fa75: Generated new TSK 0
I20260927 10:49:02.680948 31397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:02.685672 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31310
2026-09-27T10:49:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:02 dist-test-slave-wmg0 krb5kdc[31409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:02 dist-test-slave-wmg0 krb5kdc[31409](info): set up 2 sockets
Sep 27 10:49:02 dist-test-slave-wmg0 krb5kdc[31409](info): commencing operation
krb5kdc: starting...
W20260927 10:49:03.733842 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:03 dist-test-slave-wmg0 krb5kdc[31409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:03.852438 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:03.962379 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:03.962683 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:03.962777 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:03.966903 31422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:03.966990 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:03.967028 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:03.967084 31422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:03.967121 31422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:03.972383 31422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:03.973433 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:03.974507 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:03.982820 31427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:03.982842 31428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:03.982841 31430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:03.983094 31422 server_base.cc:1034] running on GCE node
I20260927 10:49:03.983758 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:03.984860 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:03.986047 31422 hybrid_clock.cc:648] HybridClock initialized: now 1790506143986032 us; error 23 us; skew 500 ppm
Sep 27 10:49:03 dist-test-slave-wmg0 krb5kdc[31409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:03.989239 31422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:03.990167 31422 webserver.cc:460] Webserver started at http://127.0.0.1:37579/ using document root <none> and password file <none>
I20260927 10:49:03.990705 31422 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:03.990763 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:03.991007 31422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:03.992949 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "a9eefb444329467db6ef1715b5c57857"
format_stamp: "Formatted at 2026-09-27 10:49:03 on dist-test-slave-wmg0"
server_key: "d95e33c895de287b8984132846849fec"
server_key_iv: "bbb9f436226f6525df4a5c373dcc49a3"
server_key_version: "encryptionkey@0"
I20260927 10:49:03.993539 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "a9eefb444329467db6ef1715b5c57857"
format_stamp: "Formatted at 2026-09-27 10:49:03 on dist-test-slave-wmg0"
server_key: "d95e33c895de287b8984132846849fec"
server_key_iv: "bbb9f436226f6525df4a5c373dcc49a3"
server_key_version: "encryptionkey@0"
I20260927 10:49:03.997027 31422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:49:03.999400 31437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:04.000607 31422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:04.000730 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "a9eefb444329467db6ef1715b5c57857"
format_stamp: "Formatted at 2026-09-27 10:49:03 on dist-test-slave-wmg0"
server_key: "d95e33c895de287b8984132846849fec"
server_key_iv: "bbb9f436226f6525df4a5c373dcc49a3"
server_key_version: "encryptionkey@0"
I20260927 10:49:04.000828 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:04.005263 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:04.008001 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:04.008179 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:04.014717 31422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35235
I20260927 10:49:04.014743 31489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35235 every 8 connection(s)
I20260927 10:49:04.015611 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:04.018128 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31422
I20260927 10:49:04.018275 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:04.018445 31490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:04.018625 10627 external_mini_cluster.cc:1428] Setting key f37419e2bff40251a3ae39026caeb5c6
I20260927 10:49:04.024073 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:04.020058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51473 (local address 127.0.0.1:35235)
0927 10:49:04.020477 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:04.020495 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:04.020538 (+ 43us) server_negotiation.cc:407] Connection header received
0927 10:49:04.021265 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:04.021287 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:04.021677 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:04.022057 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:04.022715 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:04.022720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:04.022734 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:04.022744 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:04.022804 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:04.022830 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:04.022965 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:04.023044 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 10:49:04.023184 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260927 10:49:04.025568 31490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857: Bootstrap starting.
I20260927 10:49:04.026741 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:04.027990 31490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:04.028854 31490 log.cc:824] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:04.031044 31490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857: No bootstrap required, opened a new log
I20260927 10:49:04.031152 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:04.028701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55943 (local address 127.0.0.1:35235)
0927 10:49:04.028807 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:04.028815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:04.029671 (+ 856us) server_negotiation.cc:407] Connection header received
0927 10:49:04.029871 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:04.029874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:04.029923 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:04.030007 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:04.030748 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:04.030752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:04.030754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:04.030766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:04.030794 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:04.030806 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:04.030897 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:04.030957 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 10:49:04.030993 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:49:04.034574 31490 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eefb444329467db6ef1715b5c57857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35235 } }
I20260927 10:49:04.034795 31490 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:04.034837 31490 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9eefb444329467db6ef1715b5c57857, State: Initialized, Role: FOLLOWER
I20260927 10:49:04.035197 31490 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eefb444329467db6ef1715b5c57857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35235 } }
I20260927 10:49:04.035305 31490 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:04.035347 31490 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:04.035414 31490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:04.036257 31490 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eefb444329467db6ef1715b5c57857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35235 } }
I20260927 10:49:04.036538 31490 leader_election.cc:302] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9eefb444329467db6ef1715b5c57857; no voters:
I20260927 10:49:04.036798 31490 leader_election.cc:288] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:04.036929 31500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:04.037168 31500 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [term 1 LEADER]: Becoming Leader. State: Replica: a9eefb444329467db6ef1715b5c57857, State: Running, Role: LEADER
I20260927 10:49:04.037812 31500 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eefb444329467db6ef1715b5c57857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35235 } }
I20260927 10:49:04.038107 31490 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:04.039460 31501 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9eefb444329467db6ef1715b5c57857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eefb444329467db6ef1715b5c57857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35235 } } }
I20260927 10:49:04.039582 31501 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:04.039938 31502 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9eefb444329467db6ef1715b5c57857. Latest consensus state: current_term: 1 leader_uuid: "a9eefb444329467db6ef1715b5c57857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9eefb444329467db6ef1715b5c57857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35235 } } }
I20260927 10:49:04.040028 31502 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:04.040869 31510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:04.043507 31510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:04.049204 31510 catalog_manager.cc:1347] Generated new cluster ID: 210be952eb0c4fb0a1480c1f7a499233
I20260927 10:49:04.049276 31510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:04.064929 31510 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:04.065299 31510 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:04.074203 31510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9eefb444329467db6ef1715b5c57857: Generated new TSK 0
I20260927 10:49:04.074947 31510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:04.089148 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31422
2026-09-27T10:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:04 dist-test-slave-wmg0 krb5kdc[31521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:04 dist-test-slave-wmg0 krb5kdc[31521](info): set up 2 sockets
Sep 27 10:49:04 dist-test-slave-wmg0 krb5kdc[31521](info): commencing operation
krb5kdc: starting...
W20260927 10:49:05.121807 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:05 dist-test-slave-wmg0 krb5kdc[31521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:05.244632 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:05.360486 31534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:05.360798 31534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:05.360888 31534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:05.364880 31534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:05.364989 31534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:05.365031 31534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:05.365069 31534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:05.365108 31534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:05.370380 31534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:05.371513 31534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:05.372568 31534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:05.380987 31539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:05.381057 31542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:05.381019 31540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:05.381038 31534 server_base.cc:1034] running on GCE node
I20260927 10:49:05.381923 31534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:05.382949 31534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:05.384109 31534 hybrid_clock.cc:648] HybridClock initialized: now 1790506145384077 us; error 44 us; skew 500 ppm
Sep 27 10:49:05 dist-test-slave-wmg0 krb5kdc[31521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:05.386840 31534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:05.387712 31534 webserver.cc:460] Webserver started at http://127.0.0.1:41759/ using document root <none> and password file <none>
I20260927 10:49:05.388262 31534 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:05.388320 31534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:05.388545 31534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:05.390591 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "5154fe9414e84e9a9541c179590f3a15"
format_stamp: "Formatted at 2026-09-27 10:49:05 on dist-test-slave-wmg0"
server_key: "bc18c78dfc71b0ead6f26cceb0557554"
server_key_iv: "29255b0513b98b495253ddf0ed4fcc44"
server_key_version: "encryptionkey@0"
I20260927 10:49:05.391155 31534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "5154fe9414e84e9a9541c179590f3a15"
format_stamp: "Formatted at 2026-09-27 10:49:05 on dist-test-slave-wmg0"
server_key: "bc18c78dfc71b0ead6f26cceb0557554"
server_key_iv: "29255b0513b98b495253ddf0ed4fcc44"
server_key_version: "encryptionkey@0"
I20260927 10:49:05.394834 31534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:49:05.397253 31549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:05.398551 31534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:05.398692 31534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "5154fe9414e84e9a9541c179590f3a15"
format_stamp: "Formatted at 2026-09-27 10:49:05 on dist-test-slave-wmg0"
server_key: "bc18c78dfc71b0ead6f26cceb0557554"
server_key_iv: "29255b0513b98b495253ddf0ed4fcc44"
server_key_version: "encryptionkey@0"
I20260927 10:49:05.398801 31534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:05.414160 31534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:05.417021 31534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:05.417230 31534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:05.424039 31534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45139
I20260927 10:49:05.424067 31601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45139 every 8 connection(s)
I20260927 10:49:05.424985 31534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:05.427692 31602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:05.430979 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31534
I20260927 10:49:05.431136 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:05.431515 10627 external_mini_cluster.cc:1428] Setting key 9632eda7d65b9ac0fcd846e49a7f5f7e
I20260927 10:49:05.433620 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15: Bootstrap starting.
I20260927 10:49:05.435690 31602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:05.436517 31602 log.cc:824] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:05.438402 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15: No bootstrap required, opened a new log
Sep 27 10:49:05 dist-test-slave-wmg0 krb5kdc[31521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:49:05.442245 31602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154fe9414e84e9a9541c179590f3a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:49:05.442502 31602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:05.442598 31602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5154fe9414e84e9a9541c179590f3a15, State: Initialized, Role: FOLLOWER
I20260927 10:49:05.443058 31602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154fe9414e84e9a9541c179590f3a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:49:05.443207 31602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:05.443308 31602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:05.443410 31602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:05.444516 31602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154fe9414e84e9a9541c179590f3a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:49:05.444866 31602 leader_election.cc:302] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5154fe9414e84e9a9541c179590f3a15; no voters:
I20260927 10:49:05.445194 31602 leader_election.cc:288] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:05.445317 31607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:05.445673 31607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [term 1 LEADER]: Becoming Leader. State: Replica: 5154fe9414e84e9a9541c179590f3a15, State: Running, Role: LEADER
I20260927 10:49:05.445978 31607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154fe9414e84e9a9541c179590f3a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } }
I20260927 10:49:05.446058 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:05.432910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:44307 (local address 127.0.0.1:45139)
0927 10:49:05.433325 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:05.433340 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:05.433393 (+ 53us) server_negotiation.cc:407] Connection header received
0927 10:49:05.434096 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:05.434114 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:05.434456 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:05.434779 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:05.435631 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:05.436828 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:05.437764 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:05.438242 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:05.438421 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:05.440898 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:05.440903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:05.440915 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:05.440950 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:05.442824 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:05.443367 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:05.443372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:05.443378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:05.443523 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:05.443825 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:05.443829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:05.443830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:05.444335 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:05.444560 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:05.444799 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 10:49:05.445031 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260927 10:49:05.446481 31602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:05.447574 31608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5154fe9414e84e9a9541c179590f3a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154fe9414e84e9a9541c179590f3a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 10:49:05.447693 31608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:05.447908 31608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5154fe9414e84e9a9541c179590f3a15. Latest consensus state: current_term: 1 leader_uuid: "5154fe9414e84e9a9541c179590f3a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5154fe9414e84e9a9541c179590f3a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45139 } } }
I20260927 10:49:05.447980 31608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:05.448326 31616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:05.449525 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:05.452000 31616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:05.457801 31616 catalog_manager.cc:1347] Generated new cluster ID: d5b0ab1899674b74b7b85054258ec06e
I20260927 10:49:05.457923 31616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:05.461846 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:05.452532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:37443 (local address 127.0.0.1:45139)
0927 10:49:05.452664 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:05.452668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:05.452992 (+ 324us) server_negotiation.cc:407] Connection header received
0927 10:49:05.453129 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:05.453132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:05.453182 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:05.453271 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:05.454039 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:05.454858 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:05.455927 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:05.456265 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:05.456391 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:05.458234 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:05.458238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:05.458240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:05.458265 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:05.459998 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:05.460535 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:05.460538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:05.460540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:05.460622 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:05.460965 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:05.460968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:05.460970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:05.461153 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:05.461277 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:05.461609 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 10:49:05.461703 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:49:05.466471 31616 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:05.467344 31616 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:05.472756 31616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5154fe9414e84e9a9541c179590f3a15: Generated new TSK 0
I20260927 10:49:05.473570 31616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:05.483489 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31534
W20260927 10:49:05.488657 31519 connection.cc:535] client connection to 127.0.0.1:45139 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45139): Connection reset by peer (error 104)
2026-09-27T10:49:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:05 dist-test-slave-wmg0 krb5kdc[31633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:05 dist-test-slave-wmg0 krb5kdc[31633](info): set up 2 sockets
Sep 27 10:49:05 dist-test-slave-wmg0 krb5kdc[31633](info): commencing operation
krb5kdc: starting...
W20260927 10:49:06.509730 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:06 dist-test-slave-wmg0 krb5kdc[31633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:06.629046 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:06.738502 31646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:06.738771 31646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:06.738814 31646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:06.743067 31646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:06.743132 31646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:06.743155 31646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:06.743172 31646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:06.743214 31646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:06.748519 31646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:06.749666 31646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:06.750715 31646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:06.759271 31651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:06.759253 31654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:06.759392 31646 server_base.cc:1034] running on GCE node
W20260927 10:49:06.759253 31652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:06.760152 31646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:06.761281 31646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:06.762496 31646 hybrid_clock.cc:648] HybridClock initialized: now 1790506146762485 us; error 27 us; skew 500 ppm
Sep 27 10:49:06 dist-test-slave-wmg0 krb5kdc[31633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:06.765210 31646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:06.766116 31646 webserver.cc:460] Webserver started at http://127.0.0.1:40297/ using document root <none> and password file <none>
I20260927 10:49:06.766631 31646 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:06.766690 31646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:06.766851 31646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:06.768831 31646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "60839eb1718b486db9f192c1b34926f1"
format_stamp: "Formatted at 2026-09-27 10:49:06 on dist-test-slave-wmg0"
server_key: "78f281a45b45b812bea9865bc5b3675e"
server_key_iv: "207dc431227a6931e393c6771374f6b1"
server_key_version: "encryptionkey@0"
I20260927 10:49:06.769328 31646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "60839eb1718b486db9f192c1b34926f1"
format_stamp: "Formatted at 2026-09-27 10:49:06 on dist-test-slave-wmg0"
server_key: "78f281a45b45b812bea9865bc5b3675e"
server_key_iv: "207dc431227a6931e393c6771374f6b1"
server_key_version: "encryptionkey@0"
I20260927 10:49:06.772971 31646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 10:49:06.775604 31661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:06.776899 31646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:06.777004 31646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "60839eb1718b486db9f192c1b34926f1"
format_stamp: "Formatted at 2026-09-27 10:49:06 on dist-test-slave-wmg0"
server_key: "78f281a45b45b812bea9865bc5b3675e"
server_key_iv: "207dc431227a6931e393c6771374f6b1"
server_key_version: "encryptionkey@0"
I20260927 10:49:06.777091 31646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:06.794673 31646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:06.797921 31646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:06.798086 31646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:06.805430 31646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38335
I20260927 10:49:06.805449 31713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38335 every 8 connection(s)
I20260927 10:49:06.806408 31646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:06.809079 31714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:06.814455 31714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1: Bootstrap starting.
I20260927 10:49:06.815475 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31646
I20260927 10:49:06.815623 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:06.815987 10627 external_mini_cluster.cc:1428] Setting key 52d8ab8e716f92389483ac71ef994d74
I20260927 10:49:06.817497 31714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:06.818231 31714 log.cc:824] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:06.820073 31714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1: No bootstrap required, opened a new log
I20260927 10:49:06.823534 31714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60839eb1718b486db9f192c1b34926f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260927 10:49:06.823769 31714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:06.823869 31714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60839eb1718b486db9f192c1b34926f1, State: Initialized, Role: FOLLOWER
I20260927 10:49:06.824370 31714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60839eb1718b486db9f192c1b34926f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260927 10:49:06.824559 31714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:06.824625 31714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:06.824715 31714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:06.825073 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:06.817086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:51633 (local address 127.0.0.1:38335)
0927 10:49:06.817606 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:06.817631 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:06.817666 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:06.818348 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:06.818377 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:06.818721 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:06.819050 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:06.819458 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:06.820465 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:06.821362 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:06.821926 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:06.822100 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:06.823064 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:06.823069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:06.823081 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:06.823461 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:06.823513 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:06.823534 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:06.823682 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:06.823899 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 10:49:06.824051 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260927 10:49:06.825872 31714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60839eb1718b486db9f192c1b34926f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260927 10:49:06.826224 31714 leader_election.cc:302] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60839eb1718b486db9f192c1b34926f1; no voters:
I20260927 10:49:06.826561 31714 leader_election.cc:288] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:06.826663 31719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:06.826869 31719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [term 1 LEADER]: Becoming Leader. State: Replica: 60839eb1718b486db9f192c1b34926f1, State: Running, Role: LEADER
I20260927 10:49:06.827262 31719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60839eb1718b486db9f192c1b34926f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } }
I20260927 10:49:06.827881 31714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:06.828262 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:06.829078 31720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60839eb1718b486db9f192c1b34926f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60839eb1718b486db9f192c1b34926f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } } }
I20260927 10:49:06.829221 31720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:06.829511 31721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60839eb1718b486db9f192c1b34926f1. Latest consensus state: current_term: 1 leader_uuid: "60839eb1718b486db9f192c1b34926f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60839eb1718b486db9f192c1b34926f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38335 } } }
I20260927 10:49:06.829612 31721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:06.829627 31728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:06.832850 31728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:06.835453 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:06.830501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:36091 (local address 127.0.0.1:38335)
0927 10:49:06.830608 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:06.830612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:06.830982 (+ 370us) server_negotiation.cc:407] Connection header received
0927 10:49:06.831173 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:06.831176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:06.831222 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:06.831267 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:06.831643 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:06.832382 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:06.833597 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:06.833902 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:06.834007 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:06.834983 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:06.834987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:06.834989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:06.835009 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:06.835033 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:06.835043 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:06.835152 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:06.835275 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 10:49:06.835306 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 10:49:06.838770 31728 catalog_manager.cc:1347] Generated new cluster ID: 3c143984c9b9474bb4366310c0e326db
I20260927 10:49:06.838835 31728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:06.859890 31728 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:06.861214 31728 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:06.867280 31728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60839eb1718b486db9f192c1b34926f1: Generated new TSK 0
I20260927 10:49:06.868149 31728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:06.872107 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31646
W20260927 10:49:06.877997 31631 connection.cc:535] client connection to 127.0.0.1:38335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38335): Connection reset by peer (error 104)
2026-09-27T10:49:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:06 dist-test-slave-wmg0 krb5kdc[31745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:06 dist-test-slave-wmg0 krb5kdc[31745](info): set up 2 sockets
Sep 27 10:49:06 dist-test-slave-wmg0 krb5kdc[31745](info): commencing operation
krb5kdc: starting...
W20260927 10:49:07.909786 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:07 dist-test-slave-wmg0 krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:08.036072 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:08.147147 31758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:08.147473 31758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:08.147564 31758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:08.151683 31758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:08.151793 31758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:08.151831 31758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:08.151891 31758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:08.151930 31758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:08.157502 31758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:08.158649 31758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:08.159720 31758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:08.168413 31758 server_base.cc:1034] running on GCE node
W20260927 10:49:08.168437 31763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:08.168376 31764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:08.168578 31766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:08.169215 31758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:08.170428 31758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:08.171598 31758 hybrid_clock.cc:648] HybridClock initialized: now 1790506148171569 us; error 40 us; skew 500 ppm
Sep 27 10:49:08 dist-test-slave-wmg0 krb5kdc[31745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:08.180254 31758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:08.181196 31758 webserver.cc:460] Webserver started at http://127.0.0.1:34901/ using document root <none> and password file <none>
I20260927 10:49:08.181847 31758 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:08.181943 31758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:08.182184 31758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:08.184274 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "99fba8914fb3468489d6c861c2a147c7"
format_stamp: "Formatted at 2026-09-27 10:49:08 on dist-test-slave-wmg0"
server_key: "d28a23b0b6d93ecaf5bc9fe779fca42a"
server_key_iv: "3df978ae5cc5e9f945a18309aa8f34d8"
server_key_version: "encryptionkey@0"
I20260927 10:49:08.184824 31758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "99fba8914fb3468489d6c861c2a147c7"
format_stamp: "Formatted at 2026-09-27 10:49:08 on dist-test-slave-wmg0"
server_key: "d28a23b0b6d93ecaf5bc9fe779fca42a"
server_key_iv: "3df978ae5cc5e9f945a18309aa8f34d8"
server_key_version: "encryptionkey@0"
I20260927 10:49:08.188550 31758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:08.190997 31773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:08.192261 31758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 10:49:08.192408 31758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "99fba8914fb3468489d6c861c2a147c7"
format_stamp: "Formatted at 2026-09-27 10:49:08 on dist-test-slave-wmg0"
server_key: "d28a23b0b6d93ecaf5bc9fe779fca42a"
server_key_iv: "3df978ae5cc5e9f945a18309aa8f34d8"
server_key_version: "encryptionkey@0"
I20260927 10:49:08.192520 31758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:08.197018 31758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:08.199887 31758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:08.200094 31758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:08.207118 31758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43757
I20260927 10:49:08.207144 31825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43757 every 8 connection(s)
I20260927 10:49:08.208151 31758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:08.211004 31826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:08.212584 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31758
I20260927 10:49:08.212724 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:08.213004 10627 external_mini_cluster.cc:1428] Setting key f8a0099a9cf314e0df96b5cd53d68e00
I20260927 10:49:08.217898 31826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7: Bootstrap starting.
I20260927 10:49:08.218657 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:08.214774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:39345 (local address 127.0.0.1:43757)
0927 10:49:08.215194 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:08.215210 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:08.215245 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:08.215879 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:08.215894 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:08.216183 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:08.216462 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:08.217339 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:08.217344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:08.217357 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:08.217367 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:08.217426 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:08.217445 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:08.217610 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:08.217716 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 10:49:08.217890 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":143,"threads_started":1}
I20260927 10:49:08.220182 31826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:08.221045 31826 log.cc:824] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:08.221443 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:08.223115 31826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7: No bootstrap required, opened a new log
I20260927 10:49:08.225337 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:08.223561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:34179 (local address 127.0.0.1:43757)
0927 10:49:08.223671 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:08.223674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:08.223878 (+ 204us) server_negotiation.cc:407] Connection header received
0927 10:49:08.224067 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:08.224070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:08.224110 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:08.224190 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:08.224952 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:08.224956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:08.224958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:08.224970 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:08.225003 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:08.225014 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:08.225107 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:08.225180 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 10:49:08.225220 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 10:49:08.227165 31826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fba8914fb3468489d6c861c2a147c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43757 } }
I20260927 10:49:08.227387 31826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:08.227463 31826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fba8914fb3468489d6c861c2a147c7, State: Initialized, Role: FOLLOWER
I20260927 10:49:08.227931 31826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fba8914fb3468489d6c861c2a147c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43757 } }
I20260927 10:49:08.228055 31826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:08.228139 31826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:08.228238 31826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:08.229446 31826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fba8914fb3468489d6c861c2a147c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43757 } }
I20260927 10:49:08.229826 31826 leader_election.cc:302] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99fba8914fb3468489d6c861c2a147c7; no voters:
I20260927 10:49:08.230147 31826 leader_election.cc:288] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:08.230290 31836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:08.230521 31836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [term 1 LEADER]: Becoming Leader. State: Replica: 99fba8914fb3468489d6c861c2a147c7, State: Running, Role: LEADER
I20260927 10:49:08.230907 31836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fba8914fb3468489d6c861c2a147c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43757 } }
I20260927 10:49:08.231380 31826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:08.232648 31838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99fba8914fb3468489d6c861c2a147c7. Latest consensus state: current_term: 1 leader_uuid: "99fba8914fb3468489d6c861c2a147c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fba8914fb3468489d6c861c2a147c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43757 } } }
I20260927 10:49:08.232779 31838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:08.233075 31837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99fba8914fb3468489d6c861c2a147c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99fba8914fb3468489d6c861c2a147c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43757 } } }
I20260927 10:49:08.233166 31845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:08.233193 31837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:08.236390 31845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:08.242378 31845 catalog_manager.cc:1347] Generated new cluster ID: 632eba84f3e64ad0b0f85c53aff832a2
I20260927 10:49:08.242457 31845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:08.251346 31845 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:08.251689 31845 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:08.257220 31845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99fba8914fb3468489d6c861c2a147c7: Generated new TSK 0
I20260927 10:49:08.258040 31845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:08.269979 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31758
2026-09-27T10:49:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:08 dist-test-slave-wmg0 krb5kdc[31857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:08 dist-test-slave-wmg0 krb5kdc[31857](info): set up 2 sockets
Sep 27 10:49:08 dist-test-slave-wmg0 krb5kdc[31857](info): commencing operation
krb5kdc: starting...
W20260927 10:49:09.313624 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:09 dist-test-slave-wmg0 krb5kdc[31857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:09.436499 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:09.547686 31870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:09.547942 31870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:09.547984 31870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:09.551944 31870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:09.552004 31870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:09.552026 31870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:09.552042 31870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:09.552058 31870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:09.557343 31870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:09.558497 31870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:09.559497 31870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:09.567797 31876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:09.567806 31878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:09.567849 31875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:09.567911 31870 server_base.cc:1034] running on GCE node
I20260927 10:49:09.568598 31870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:09.569756 31870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:09.570966 31870 hybrid_clock.cc:648] HybridClock initialized: now 1790506149570929 us; error 51 us; skew 500 ppm
Sep 27 10:49:09 dist-test-slave-wmg0 krb5kdc[31857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:09.573752 31870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:09.574679 31870 webserver.cc:460] Webserver started at http://127.0.0.1:36189/ using document root <none> and password file <none>
I20260927 10:49:09.575239 31870 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:09.575327 31870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:09.575556 31870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:09.577711 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "4b702111fb914f168c96dfc5cdd5c81e"
format_stamp: "Formatted at 2026-09-27 10:49:09 on dist-test-slave-wmg0"
server_key: "a57452d3736e18b2f2ffb5d9a026d6ed"
server_key_iv: "6ee04550c61d87f7a215b882ca72cc7f"
server_key_version: "encryptionkey@0"
I20260927 10:49:09.578277 31870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "4b702111fb914f168c96dfc5cdd5c81e"
format_stamp: "Formatted at 2026-09-27 10:49:09 on dist-test-slave-wmg0"
server_key: "a57452d3736e18b2f2ffb5d9a026d6ed"
server_key_iv: "6ee04550c61d87f7a215b882ca72cc7f"
server_key_version: "encryptionkey@0"
I20260927 10:49:09.582007 31870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:49:09.584395 31885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:09.585697 31870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:09.585836 31870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "4b702111fb914f168c96dfc5cdd5c81e"
format_stamp: "Formatted at 2026-09-27 10:49:09 on dist-test-slave-wmg0"
server_key: "a57452d3736e18b2f2ffb5d9a026d6ed"
server_key_iv: "6ee04550c61d87f7a215b882ca72cc7f"
server_key_version: "encryptionkey@0"
I20260927 10:49:09.585954 31870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:09.621650 31870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:09.625136 31870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:09.625351 31870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:09.632766 31870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38619
I20260927 10:49:09.632783 31937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38619 every 8 connection(s)
I20260927 10:49:09.633806 31870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:09.636548 31938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:09.642083 31938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e: Bootstrap starting.
I20260927 10:49:09.643323 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31870
I20260927 10:49:09.643460 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:09.643729 10627 external_mini_cluster.cc:1428] Setting key 8f5e78f959443298d8d59ff38a0cfcc7
I20260927 10:49:09.644682 31938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:09.645581 31938 log.cc:824] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:09.647619 31938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e: No bootstrap required, opened a new log
I20260927 10:49:09.651021 31938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b702111fb914f168c96dfc5cdd5c81e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260927 10:49:09.651260 31938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:09.651341 31938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b702111fb914f168c96dfc5cdd5c81e, State: Initialized, Role: FOLLOWER
I20260927 10:49:09.651782 31938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b702111fb914f168c96dfc5cdd5c81e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260927 10:49:09.651947 31938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:09.652021 31938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:09.652122 31938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 0 FOLLOWER]: Advancing to term 1
Sep 27 10:49:09 dist-test-slave-wmg0 krb5kdc[31857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 10:49:09.653326 31938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b702111fb914f168c96dfc5cdd5c81e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260927 10:49:09.653717 31938 leader_election.cc:302] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b702111fb914f168c96dfc5cdd5c81e; no voters:
I20260927 10:49:09.654101 31938 leader_election.cc:288] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:09.654206 31943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:09.654532 31943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [term 1 LEADER]: Becoming Leader. State: Replica: 4b702111fb914f168c96dfc5cdd5c81e, State: Running, Role: LEADER
I20260927 10:49:09.655124 31943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b702111fb914f168c96dfc5cdd5c81e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } }
I20260927 10:49:09.655421 31938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:09.657145 31944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b702111fb914f168c96dfc5cdd5c81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b702111fb914f168c96dfc5cdd5c81e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } } }
I20260927 10:49:09.657280 31944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:09.657671 31957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:09.657650 31945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b702111fb914f168c96dfc5cdd5c81e. Latest consensus state: current_term: 1 leader_uuid: "4b702111fb914f168c96dfc5cdd5c81e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b702111fb914f168c96dfc5cdd5c81e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38619 } } }
I20260927 10:49:09.657986 31945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:09.659870 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:09.644864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:42003 (local address 127.0.0.1:38619)
0927 10:49:09.645310 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:09.645325 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:09.645362 (+ 37us) server_negotiation.cc:407] Connection header received
0927 10:49:09.646149 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:09.646168 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:09.646593 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:09.646936 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:09.647767 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:09.648731 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:09.649651 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:09.650131 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:09.650302 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:09.653456 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:09.653465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:09.653514 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:09.653554 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:09.655984 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:09.656691 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:09.656698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:09.656706 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:09.656871 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:09.657329 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:09.657332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:09.657334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:09.657917 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:09.658117 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:09.658883 (+ 766us) server_negotiation.cc:299] Negotiation successful
0927 10:49:09.659133 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260927 10:49:09.660697 31957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:09.664124 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:09.668459 31957 catalog_manager.cc:1347] Generated new cluster ID: 6a207ccaa3da4ee388e8e7d9731f7282
I20260927 10:49:09.668576 31957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:09.675035 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:09.666310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:47645 (local address 127.0.0.1:38619)
0927 10:49:09.666423 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:09.666428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:09.666666 (+ 238us) server_negotiation.cc:407] Connection header received
0927 10:49:09.666784 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:09.666786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:09.666821 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:09.666920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:09.667593 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:09.668176 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:09.669256 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:09.669675 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:09.669786 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:09.671345 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:09.671349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:09.671366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:09.671390 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:09.673085 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:09.673593 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:09.673597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:09.673599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:09.673684 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:09.674016 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:09.674020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:09.674022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:09.674226 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:09.674384 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:09.674612 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 10:49:09.674873 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:49:09.683100 31957 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:09.684150 31957 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:09.696018 31957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b702111fb914f168c96dfc5cdd5c81e: Generated new TSK 0
I20260927 10:49:09.696841 31957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:09.704907 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31870
W20260927 10:49:09.710781 31855 connection.cc:535] client connection to 127.0.0.1:38619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38619): Connection reset by peer (error 104)
2026-09-27T10:49:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:09 dist-test-slave-wmg0 krb5kdc[31969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:09 dist-test-slave-wmg0 krb5kdc[31969](info): set up 2 sockets
Sep 27 10:49:09 dist-test-slave-wmg0 krb5kdc[31969](info): commencing operation
krb5kdc: starting...
W20260927 10:49:10.733732 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:10 dist-test-slave-wmg0 krb5kdc[31969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:10.852284 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:10.968992 31982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:10.969251 31982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:10.969293 31982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:10.973325 31982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:10.973380 31982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:10.973399 31982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:10.973416 31982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:10.973456 31982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:10.979893 31982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:10.980964 31982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:10.982084 31982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:10.990451 31990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:10.990473 31987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:10.990442 31988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:10.990815 31982 server_base.cc:1034] running on GCE node
I20260927 10:49:10.991382 31982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:10.992406 31982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:10.993543 31982 hybrid_clock.cc:648] HybridClock initialized: now 1790506150993508 us; error 42 us; skew 500 ppm
Sep 27 10:49:10 dist-test-slave-wmg0 krb5kdc[31969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:10.996296 31982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:10.997191 31982 webserver.cc:460] Webserver started at http://127.0.0.1:37373/ using document root <none> and password file <none>
I20260927 10:49:10.997802 31982 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:10.997862 31982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:10.998114 31982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:11.000101 31982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "ba647d3e33a24eeab04fd0d6bae843d8"
format_stamp: "Formatted at 2026-09-27 10:49:10 on dist-test-slave-wmg0"
server_key: "a2cd10532811ac454da387cf2945a702"
server_key_iv: "5c1497a6aaba16480fe3272f4ee164dd"
server_key_version: "encryptionkey@0"
I20260927 10:49:11.000658 31982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "ba647d3e33a24eeab04fd0d6bae843d8"
format_stamp: "Formatted at 2026-09-27 10:49:10 on dist-test-slave-wmg0"
server_key: "a2cd10532811ac454da387cf2945a702"
server_key_iv: "5c1497a6aaba16480fe3272f4ee164dd"
server_key_version: "encryptionkey@0"
I20260927 10:49:11.004369 31982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 10:49:11.006834 31997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:11.008320 31982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 10:49:11.008468 31982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "ba647d3e33a24eeab04fd0d6bae843d8"
format_stamp: "Formatted at 2026-09-27 10:49:10 on dist-test-slave-wmg0"
server_key: "a2cd10532811ac454da387cf2945a702"
server_key_iv: "5c1497a6aaba16480fe3272f4ee164dd"
server_key_version: "encryptionkey@0"
I20260927 10:49:11.008577 31982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:11.021718 31982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:11.024957 31982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:11.025174 31982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:11.032511 31982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39079
I20260927 10:49:11.032533 32049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39079 every 8 connection(s)
I20260927 10:49:11.033547 31982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:11.036213 32050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:11.038878 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 31982
I20260927 10:49:11.039031 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:11.039355 10627 external_mini_cluster.cc:1428] Setting key 88e73a79023b866f6789ade5036f8d28
I20260927 10:49:11.042395 32050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8: Bootstrap starting.
I20260927 10:49:11.044927 32050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:11.045732 32050 log.cc:824] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:11.047518 32050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8: No bootstrap required, opened a new log
I20260927 10:49:11.048250 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:11.040819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:54745 (local address 127.0.0.1:39079)
0927 10:49:11.041258 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:11.041273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:11.041307 (+ 34us) server_negotiation.cc:407] Connection header received
0927 10:49:11.041958 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:11.041975 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:11.042309 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:11.042655 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:11.043090 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:11.044120 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:11.045002 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:11.045527 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:11.045697 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:11.046618 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:11.046623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:11.046635 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:11.047018 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:11.047069 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:11.047091 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:11.047230 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:11.047398 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 10:49:11.047529 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":126,"threads_started":1}
I20260927 10:49:11.051329 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:11.051666 32050 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260927 10:49:11.051878 32050 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:11.051956 32050 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba647d3e33a24eeab04fd0d6bae843d8, State: Initialized, Role: FOLLOWER
I20260927 10:49:11.052368 32050 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260927 10:49:11.052520 32050 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:11.052647 32050 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:11.052763 32050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:11.053817 32050 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260927 10:49:11.054209 32050 leader_election.cc:302] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba647d3e33a24eeab04fd0d6bae843d8; no voters:
I20260927 10:49:11.054530 32050 leader_election.cc:288] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:11.054765 32060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:11.055078 32060 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [term 1 LEADER]: Becoming Leader. State: Replica: ba647d3e33a24eeab04fd0d6bae843d8, State: Running, Role: LEADER
I20260927 10:49:11.055409 32060 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } }
I20260927 10:49:11.055792 32050 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:11.057675 32061 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } } }
I20260927 10:49:11.057821 32061 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:11.058959 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:11.053446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:47235 (local address 127.0.0.1:39079)
0927 10:49:11.053586 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:11.053590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:11.053716 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:49:11.053810 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:11.053813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:11.053862 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:11.053951 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:11.054217 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:11.055002 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:11.056112 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:11.056479 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:11.056584 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:11.058350 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:11.058353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:11.058356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:11.058369 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:11.058396 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:11.058408 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:11.058509 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:11.058811 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 10:49:11.058846 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 10:49:11.058365 32062 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba647d3e33a24eeab04fd0d6bae843d8. Latest consensus state: current_term: 1 leader_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba647d3e33a24eeab04fd0d6bae843d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39079 } } }
I20260927 10:49:11.059099 32062 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8 [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:11.061055 32074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:11.061121 32074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:11.061527 32075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:11.062209 32075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:11.067140 32075 catalog_manager.cc:1347] Generated new cluster ID: 7b279e8d9f9f45f1a27d5bc37ac24224
I20260927 10:49:11.067199 32075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:11.095355 32075 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:11.096678 32075 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:11.106302 32075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba647d3e33a24eeab04fd0d6bae843d8: Generated new TSK 0
I20260927 10:49:11.107110 32075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:11.117594 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 31982
W20260927 10:49:11.122743 31967 connection.cc:535] client connection to 127.0.0.1:39079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39079): Connection reset by peer (error 104)
2026-09-27T10:49:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:11 dist-test-slave-wmg0 krb5kdc[32081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:11 dist-test-slave-wmg0 krb5kdc[32081](info): set up 2 sockets
Sep 27 10:49:11 dist-test-slave-wmg0 krb5kdc[32081](info): commencing operation
krb5kdc: starting...
W20260927 10:49:12.157753 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:12 dist-test-slave-wmg0 krb5kdc[32081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 10:49:12.279843 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:12.387713 32094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:12.388016 32094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:12.388089 32094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:12.392088 32094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:12.392181 32094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:12.392222 32094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:12.392283 32094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:12.392323 32094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:12.397477 32094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:12.398522 32094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:12.399524 32094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:12.407752 32102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:12.407868 32094 server_base.cc:1034] running on GCE node
W20260927 10:49:12.407825 32100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:12.408061 32099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:12.408669 32094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:12.409817 32094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:12.410983 32094 hybrid_clock.cc:648] HybridClock initialized: now 1790506152410961 us; error 43 us; skew 500 ppm
Sep 27 10:49:12 dist-test-slave-wmg0 krb5kdc[32081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:12.413702 32094 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 10:49:12.414536 32094 webserver.cc:460] Webserver started at http://127.0.0.1:45617/ using document root <none> and password file <none>
I20260927 10:49:12.415112 32094 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:12.415175 32094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:12.415375 32094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:12.417346 32094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "bcb7d02a8ffe415f81274649654b115e"
format_stamp: "Formatted at 2026-09-27 10:49:12 on dist-test-slave-wmg0"
server_key: "8f776f9178eded9d5a2db99e7be0043e"
server_key_iv: "be8dd54217d6d643ad14a27842edef04"
server_key_version: "encryptionkey@0"
I20260927 10:49:12.417963 32094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "bcb7d02a8ffe415f81274649654b115e"
format_stamp: "Formatted at 2026-09-27 10:49:12 on dist-test-slave-wmg0"
server_key: "8f776f9178eded9d5a2db99e7be0043e"
server_key_iv: "be8dd54217d6d643ad14a27842edef04"
server_key_version: "encryptionkey@0"
I20260927 10:49:12.421677 32094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 10:49:12.423985 32109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:12.425184 32094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:12.425320 32094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "bcb7d02a8ffe415f81274649654b115e"
format_stamp: "Formatted at 2026-09-27 10:49:12 on dist-test-slave-wmg0"
server_key: "8f776f9178eded9d5a2db99e7be0043e"
server_key_iv: "be8dd54217d6d643ad14a27842edef04"
server_key_version: "encryptionkey@0"
I20260927 10:49:12.425441 32094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:12.429752 32094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:12.432548 32094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:12.432744 32094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:12.439805 32094 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43037
I20260927 10:49:12.439854 32161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43037 every 8 connection(s)
I20260927 10:49:12.440709 32094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:12.443436 32162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:12.445684 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32094
I20260927 10:49:12.445830 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:12.446146 10627 external_mini_cluster.cc:1428] Setting key a55d45bb52c7c7b7700793b451ca2e14
I20260927 10:49:12.448999 32162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e: Bootstrap starting.
I20260927 10:49:12.451182 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:12.447512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:40295 (local address 127.0.0.1:43037)
0927 10:49:12.447925 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:12.447946 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:12.447981 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:12.448599 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:12.448616 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:12.448906 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:12.449236 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:12.450047 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:12.450052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:12.450073 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:12.450084 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:12.450134 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:12.450152 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:12.450302 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:12.450372 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 10:49:12.450539 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":130,"threads_started":1}
I20260927 10:49:12.451350 32162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:12.452229 32162 log.cc:824] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:12.454088 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:12.454375 32162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e: No bootstrap required, opened a new log
I20260927 10:49:12.457777 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:12.456057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:34301 (local address 127.0.0.1:43037)
0927 10:49:12.456177 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:12.456181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:12.456431 (+ 250us) server_negotiation.cc:407] Connection header received
0927 10:49:12.456521 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:12.456524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:12.456552 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:12.456615 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:12.457325 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:12.457327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:12.457329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:12.457339 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:12.457360 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:12.457369 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:12.457447 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:12.457595 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 10:49:12.457628 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:49:12.458581 32162 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7d02a8ffe415f81274649654b115e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43037 } }
I20260927 10:49:12.458815 32162 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:12.458880 32162 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb7d02a8ffe415f81274649654b115e, State: Initialized, Role: FOLLOWER
I20260927 10:49:12.459312 32162 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7d02a8ffe415f81274649654b115e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43037 } }
I20260927 10:49:12.459460 32162 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:12.459523 32162 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:12.459626 32162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:12.460475 32162 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7d02a8ffe415f81274649654b115e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43037 } }
I20260927 10:49:12.460799 32162 leader_election.cc:302] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcb7d02a8ffe415f81274649654b115e; no voters:
I20260927 10:49:12.461084 32162 leader_election.cc:288] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:12.461200 32172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:12.461428 32172 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [term 1 LEADER]: Becoming Leader. State: Replica: bcb7d02a8ffe415f81274649654b115e, State: Running, Role: LEADER
I20260927 10:49:12.461807 32172 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7d02a8ffe415f81274649654b115e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43037 } }
I20260927 10:49:12.462270 32162 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:12.463593 32173 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcb7d02a8ffe415f81274649654b115e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7d02a8ffe415f81274649654b115e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43037 } } }
I20260927 10:49:12.463735 32173 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:12.464730 32174 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb7d02a8ffe415f81274649654b115e. Latest consensus state: current_term: 1 leader_uuid: "bcb7d02a8ffe415f81274649654b115e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb7d02a8ffe415f81274649654b115e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43037 } } }
I20260927 10:49:12.464869 32174 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:12.466781 32187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:12.466845 32187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:12.466918 32183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:12.467593 32183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:12.472455 32183 catalog_manager.cc:1347] Generated new cluster ID: b106190f529b4ba1a45adf461e15a483
I20260927 10:49:12.472513 32183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:12.494736 32183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:12.495087 32183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:12.503213 32183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcb7d02a8ffe415f81274649654b115e: Generated new TSK 0
I20260927 10:49:12.503911 32183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:12.511467 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32094
2026-09-27T10:49:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:12 dist-test-slave-wmg0 krb5kdc[32193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:12 dist-test-slave-wmg0 krb5kdc[32193](info): set up 2 sockets
Sep 27 10:49:12 dist-test-slave-wmg0 krb5kdc[32193](info): commencing operation
krb5kdc: starting...
W20260927 10:49:13.541743 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:13 dist-test-slave-wmg0 krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:13.658403 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36807
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:36807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:13.766360 32206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:13.766633 32206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:13.766675 32206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:13.770643 32206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:13.770705 32206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:13.770725 32206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:13.770741 32206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:13.770757 32206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:13.775905 32206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:36807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:13.776921 32206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:13.777920 32206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:13.786257 32214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:13.786255 32212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:13.786384 32206 server_base.cc:1034] running on GCE node
W20260927 10:49:13.786227 32211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:13.787093 32206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:13.788116 32206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:13.789271 32206 hybrid_clock.cc:648] HybridClock initialized: now 1790506153789235 us; error 49 us; skew 500 ppm
Sep 27 10:49:13 dist-test-slave-wmg0 krb5kdc[32193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506153, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:13.796026 32206 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:13.796990 32206 webserver.cc:460] Webserver started at http://127.10.96.254:45371/ using document root <none> and password file <none>
I20260927 10:49:13.797624 32206 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:13.797716 32206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:13.797943 32206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:13.799975 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "22d0f23f4bb749969548b512e8ec259b"
format_stamp: "Formatted at 2026-09-27 10:49:13 on dist-test-slave-wmg0"
server_key: "ce83a8bb26a3fbdea6a79761991d92aa"
server_key_iv: "6cc7922bd37bc6935b8db07043940d46"
server_key_version: "encryptionkey@0"
I20260927 10:49:13.800534 32206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "22d0f23f4bb749969548b512e8ec259b"
format_stamp: "Formatted at 2026-09-27 10:49:13 on dist-test-slave-wmg0"
server_key: "ce83a8bb26a3fbdea6a79761991d92aa"
server_key_iv: "6cc7922bd37bc6935b8db07043940d46"
server_key_version: "encryptionkey@0"
I20260927 10:49:13.804183 32206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:13.806659 32221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:13.807951 32206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:13.808106 32206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "22d0f23f4bb749969548b512e8ec259b"
format_stamp: "Formatted at 2026-09-27 10:49:13 on dist-test-slave-wmg0"
server_key: "ce83a8bb26a3fbdea6a79761991d92aa"
server_key_iv: "6cc7922bd37bc6935b8db07043940d46"
server_key_version: "encryptionkey@0"
I20260927 10:49:13.808220 32206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:13.841768 32206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:13.845021 32206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:13.845230 32206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:13.852459 32206 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:36807
I20260927 10:49:13.852476 32273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:36807 every 8 connection(s)
I20260927 10:49:13.853426 32206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:13.854465 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32206
I20260927 10:49:13.854614 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:13.854912 10627 external_mini_cluster.cc:1428] Setting key e4a982910c89d1f48c8dbd4bb337b880
I20260927 10:49:13.856547 32274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:13.861960 32274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b: Bootstrap starting.
Sep 27 10:49:13 dist-test-slave-wmg0 krb5kdc[32193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:13.864662 32274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:13.865566 32274 log.cc:824] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:13.867426 32274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b: No bootstrap required, opened a new log
I20260927 10:49:13.869318 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:13.856518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57597 (local address 127.10.96.254:36807)
0927 10:49:13.856939 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:13.856954 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:13.856990 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:49:13.857712 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:13.857730 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:13.858065 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:13.858407 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:13.859200 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:13.860210 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:13.861101 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:13.861632 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:13.861801 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:13.864568 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:13.864573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:13.864587 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:13.864613 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:13.866445 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:13.866978 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:13.866983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:13.866987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:13.867087 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:13.867430 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:13.867436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:13.867437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:13.867865 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:13.868074 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:13.868412 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:49:13.868630 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":133,"threads_started":1}
I20260927 10:49:13.871307 32274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d0f23f4bb749969548b512e8ec259b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36807 } }
I20260927 10:49:13.871594 32274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:13.871675 32274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d0f23f4bb749969548b512e8ec259b, State: Initialized, Role: FOLLOWER
I20260927 10:49:13.872146 32274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d0f23f4bb749969548b512e8ec259b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36807 } }
I20260927 10:49:13.872298 32274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:13.872372 32274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:13.872483 32274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:13.872623 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:13.873447 32274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d0f23f4bb749969548b512e8ec259b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36807 } }
I20260927 10:49:13.873886 32274 leader_election.cc:302] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22d0f23f4bb749969548b512e8ec259b; no voters:
I20260927 10:49:13.874238 32274 leader_election.cc:288] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:13.874302 32283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:13.874531 32283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [term 1 LEADER]: Becoming Leader. State: Replica: 22d0f23f4bb749969548b512e8ec259b, State: Running, Role: LEADER
I20260927 10:49:13.874871 32283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d0f23f4bb749969548b512e8ec259b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36807 } }
I20260927 10:49:13.875519 32274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:13.876602 32284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22d0f23f4bb749969548b512e8ec259b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d0f23f4bb749969548b512e8ec259b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36807 } } }
I20260927 10:49:13.876731 32284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:13.877048 32285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d0f23f4bb749969548b512e8ec259b. Latest consensus state: current_term: 1 leader_uuid: "22d0f23f4bb749969548b512e8ec259b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d0f23f4bb749969548b512e8ec259b" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36807 } } }
I20260927 10:49:13.877122 32293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:13.877195 32285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:13.880398 32293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:13.883764 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:13.874996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48439 (local address 127.10.96.254:36807)
0927 10:49:13.875088 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:13.875092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:13.875301 (+ 209us) server_negotiation.cc:407] Connection header received
0927 10:49:13.875434 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:13.875437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:13.875485 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:13.875568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:13.876407 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:13.877247 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:13.878448 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:13.878688 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:13.878798 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:13.880445 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:13.880449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:13.880456 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:13.880484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:13.882046 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:13.882515 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:13.882519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:13.882520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:13.882605 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:13.882933 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:13.882936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:13.882938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:13.883112 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:13.883259 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:13.883516 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 10:49:13.883607 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:49:13.886667 32293 catalog_manager.cc:1347] Generated new cluster ID: cf12707ff03a498ea849bdebf027f155
I20260927 10:49:13.886727 32293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:13.906309 32293 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:13.907390 32293 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:13.912611 32293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22d0f23f4bb749969548b512e8ec259b: Generated new TSK 0
I20260927 10:49:13.913347 32293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:13.918912 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32206
W20260927 10:49:13.924170 32191 connection.cc:535] client connection to 127.10.96.254:36807 recv error: Network error: failed to read from TLS socket (remote: 127.10.96.254:36807): Connection reset by peer (error 104)
2026-09-27T10:49:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:13 dist-test-slave-wmg0 krb5kdc[32305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:13 dist-test-slave-wmg0 krb5kdc[32305](info): set up 2 sockets
Sep 27 10:49:13 dist-test-slave-wmg0 krb5kdc[32305](info): commencing operation
krb5kdc: starting...
W20260927 10:49:14.945861 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:15 dist-test-slave-wmg0 krb5kdc[32305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:15.070147 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37381
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:15.179509 32318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:15.179795 32318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:15.179874 32318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:15.183859 32318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:15.183944 32318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:15.183997 32318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:15.184052 32318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:15.184090 32318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:15.189687 32318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:37381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:15.190831 32318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:15.191917 32318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:15.201037 32324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:15.201102 32318 server_base.cc:1034] running on GCE node
W20260927 10:49:15.201033 32326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:15.201252 32323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:15.201903 32318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:15.203049 32318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:15.204224 32318 hybrid_clock.cc:648] HybridClock initialized: now 1790506155204197 us; error 44 us; skew 500 ppm
Sep 27 10:49:15 dist-test-slave-wmg0 krb5kdc[32305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:15.207228 32318 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:15.208137 32318 webserver.cc:460] Webserver started at http://127.10.96.254:36275/ using document root <none> and password file <none>
I20260927 10:49:15.208709 32318 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:15.208798 32318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:15.209028 32318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:15.211143 32318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "f88f287b0c0a48568561ea087b40072c"
format_stamp: "Formatted at 2026-09-27 10:49:15 on dist-test-slave-wmg0"
server_key: "cfe0be46aba888b58c432dfa20fcade9"
server_key_iv: "3f485df74091dfec9678f53ffa8d2d14"
server_key_version: "encryptionkey@0"
I20260927 10:49:15.211699 32318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "f88f287b0c0a48568561ea087b40072c"
format_stamp: "Formatted at 2026-09-27 10:49:15 on dist-test-slave-wmg0"
server_key: "cfe0be46aba888b58c432dfa20fcade9"
server_key_iv: "3f485df74091dfec9678f53ffa8d2d14"
server_key_version: "encryptionkey@0"
I20260927 10:49:15.215519 32318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260927 10:49:15.218004 32333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:15.219301 32318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:15.219448 32318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "f88f287b0c0a48568561ea087b40072c"
format_stamp: "Formatted at 2026-09-27 10:49:15 on dist-test-slave-wmg0"
server_key: "cfe0be46aba888b58c432dfa20fcade9"
server_key_iv: "3f485df74091dfec9678f53ffa8d2d14"
server_key_version: "encryptionkey@0"
I20260927 10:49:15.219559 32318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:15.230288 32318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:15.233249 32318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:15.233503 32318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:15.240540 32318 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:37381
I20260927 10:49:15.240568 32385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:37381 every 8 connection(s)
I20260927 10:49:15.241513 32318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:15.244154 32386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:15.246248 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32318
I20260927 10:49:15.246392 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:15.246687 10627 external_mini_cluster.cc:1428] Setting key e5ca946c8182a29fa66907d00ad687c3
I20260927 10:49:15.250926 32386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c: Bootstrap starting.
I20260927 10:49:15.253153 32386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:15.254251 32386 log.cc:824] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:15.256193 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:15.248124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39907 (local address 127.10.96.254:37381)
0927 10:49:15.248577 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:15.248593 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:15.248628 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:15.249297 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:15.249315 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:15.249718 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:15.250061 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:15.250459 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:15.251480 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:15.252387 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:15.252874 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:15.253344 (+ 470us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:15.254298 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:15.254302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:15.254314 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:15.254845 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:15.254913 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:15.254946 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:15.255087 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:15.255318 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 10:49:15.255459 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":127,"threads_started":1}
I20260927 10:49:15.256332 32386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c: No bootstrap required, opened a new log
I20260927 10:49:15.258961 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:15.260233 32386 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88f287b0c0a48568561ea087b40072c" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37381 } }
I20260927 10:49:15.260515 32386 raft_consensus.cc:374] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:15.260577 32386 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88f287b0c0a48568561ea087b40072c, State: Initialized, Role: FOLLOWER
I20260927 10:49:15.261036 32386 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88f287b0c0a48568561ea087b40072c" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37381 } }
I20260927 10:49:15.261207 32386 raft_consensus.cc:388] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:15.261322 32386 raft_consensus.cc:482] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:15.261442 32386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:15.262538 32386 raft_consensus.cc:504] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88f287b0c0a48568561ea087b40072c" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37381 } }
I20260927 10:49:15.263231 32386 leader_election.cc:302] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f88f287b0c0a48568561ea087b40072c; no voters:
I20260927 10:49:15.263548 32386 leader_election.cc:288] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:15.263938 32397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:15.264325 32397 raft_consensus.cc:686] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [term 1 LEADER]: Becoming Leader. State: Replica: f88f287b0c0a48568561ea087b40072c, State: Running, Role: LEADER
I20260927 10:49:15.264926 32386 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:15.264889 32397 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88f287b0c0a48568561ea087b40072c" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37381 } }
I20260927 10:49:15.266189 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:15.261165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57553 (local address 127.10.96.254:37381)
0927 10:49:15.261273 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:15.261277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:15.261506 (+ 229us) server_negotiation.cc:407] Connection header received
0927 10:49:15.261685 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:15.261687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:15.261725 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:15.261783 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:15.262073 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:15.262663 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:15.263915 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:15.264251 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:15.264325 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:15.265492 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:15.265499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:15.265502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:15.265514 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:15.265540 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:15.265559 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:15.265675 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:15.265941 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 10:49:15.266011 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:49:15.267334 32396 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f88f287b0c0a48568561ea087b40072c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88f287b0c0a48568561ea087b40072c" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37381 } } }
I20260927 10:49:15.267489 32396 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:15.267854 32398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88f287b0c0a48568561ea087b40072c. Latest consensus state: current_term: 1 leader_uuid: "f88f287b0c0a48568561ea087b40072c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88f287b0c0a48568561ea087b40072c" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37381 } } }
I20260927 10:49:15.268031 32398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:15.271857 32411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:15.271915 32411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:15.272003 32405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:15.272784 32405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:15.282542 32405 catalog_manager.cc:1347] Generated new cluster ID: 453a75737f7f462dad7b875fe05c935b
I20260927 10:49:15.282640 32405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:15.300963 32405 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:15.302028 32405 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:15.322175 32405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f88f287b0c0a48568561ea087b40072c: Generated new TSK 0
I20260927 10:49:15.323033 32405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:15.332998 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32318
W20260927 10:49:15.338901 32303 connection.cc:535] client connection to 127.10.96.254:37381 recv error: Network error: failed to read from TLS socket (remote: 127.10.96.254:37381): Connection reset by peer (error 104)
2026-09-27T10:49:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:15 dist-test-slave-wmg0 krb5kdc[32417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:15 dist-test-slave-wmg0 krb5kdc[32417](info): set up 2 sockets
Sep 27 10:49:15 dist-test-slave-wmg0 krb5kdc[32417](info): commencing operation
krb5kdc: starting...
W20260927 10:49:16.361909 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:16 dist-test-slave-wmg0 krb5kdc[32417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:16.480962 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43783
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:43783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:16.592648 32430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:16.592901 32430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:16.592942 32430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:16.596922 32430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:16.596987 32430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:16.597007 32430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:16.597023 32430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:16.597038 32430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:16.602805 32430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:43783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:16.603901 32430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:16.604955 32430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:16.613638 32430 server_base.cc:1034] running on GCE node
W20260927 10:49:16.613615 32435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:16.613615 32438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:16.613657 32436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:16.614574 32430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:16.615746 32430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:16.616948 32430 hybrid_clock.cc:648] HybridClock initialized: now 1790506156616918 us; error 44 us; skew 500 ppm
Sep 27 10:49:16 dist-test-slave-wmg0 krb5kdc[32417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:16.619800 32430 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:16.620743 32430 webserver.cc:460] Webserver started at http://127.10.96.254:44475/ using document root <none> and password file <none>
I20260927 10:49:16.621294 32430 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:16.621383 32430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:16.621634 32430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:16.623636 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "96fa3df9d86748728542151c2eee7138"
format_stamp: "Formatted at 2026-09-27 10:49:16 on dist-test-slave-wmg0"
server_key: "4c1c9642e15c6361436ea9256f3de7b5"
server_key_iv: "5fab9af2307f578dbf389c460ac900d3"
server_key_version: "encryptionkey@0"
I20260927 10:49:16.624203 32430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "96fa3df9d86748728542151c2eee7138"
format_stamp: "Formatted at 2026-09-27 10:49:16 on dist-test-slave-wmg0"
server_key: "4c1c9642e15c6361436ea9256f3de7b5"
server_key_iv: "5fab9af2307f578dbf389c460ac900d3"
server_key_version: "encryptionkey@0"
I20260927 10:49:16.627954 32430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 10:49:16.630661 32445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:16.631989 32430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 10:49:16.632140 32430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "96fa3df9d86748728542151c2eee7138"
format_stamp: "Formatted at 2026-09-27 10:49:16 on dist-test-slave-wmg0"
server_key: "4c1c9642e15c6361436ea9256f3de7b5"
server_key_iv: "5fab9af2307f578dbf389c460ac900d3"
server_key_version: "encryptionkey@0"
I20260927 10:49:16.632256 32430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:16.636859 32430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:16.640100 32430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:16.640290 32430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:16.647404 32430 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:43783
I20260927 10:49:16.647421 32497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:43783 every 8 connection(s)
I20260927 10:49:16.648365 32430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:16.651049 32498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:16.656494 32498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138: Bootstrap starting.
I20260927 10:49:16.657196 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32430
I20260927 10:49:16.657339 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:16.657671 10627 external_mini_cluster.cc:1428] Setting key 6636bc68cb76494b6944830f4517cd9f
I20260927 10:49:16.659507 32498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:16.660228 32498 log.cc:824] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:16.662377 32498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138: No bootstrap required, opened a new log
I20260927 10:49:16.663390 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:16.658953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37649 (local address 127.10.96.254:43783)
0927 10:49:16.659524 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:16.659538 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:16.659705 (+ 167us) server_negotiation.cc:407] Connection header received
0927 10:49:16.660448 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:16.660468 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:16.660823 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:16.661217 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:16.662027 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:16.662033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:16.662055 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:16.662065 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:16.662146 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:16.662175 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:16.662317 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:16.662443 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 10:49:16.662711 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":139,"threads_started":1}
I20260927 10:49:16.666083 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:16.666111 32498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fa3df9d86748728542151c2eee7138" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43783 } }
I20260927 10:49:16.666388 32498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:16.666443 32498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fa3df9d86748728542151c2eee7138, State: Initialized, Role: FOLLOWER
I20260927 10:49:16.666843 32498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fa3df9d86748728542151c2eee7138" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43783 } }
I20260927 10:49:16.666996 32498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:16.667083 32498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:16.667182 32498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:16.668179 32498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fa3df9d86748728542151c2eee7138" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43783 } }
I20260927 10:49:16.668545 32498 leader_election.cc:302] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96fa3df9d86748728542151c2eee7138; no voters:
I20260927 10:49:16.668874 32498 leader_election.cc:288] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:16.668958 32508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:16.669452 32508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [term 1 LEADER]: Becoming Leader. State: Replica: 96fa3df9d86748728542151c2eee7138, State: Running, Role: LEADER
I20260927 10:49:16.669834 32508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fa3df9d86748728542151c2eee7138" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43783 } }
I20260927 10:49:16.670130 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:16.668279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45185 (local address 127.10.96.254:43783)
0927 10:49:16.668386 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:16.668389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:16.668565 (+ 176us) server_negotiation.cc:407] Connection header received
0927 10:49:16.668704 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:16.668707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:16.668740 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:16.668831 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:16.669593 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:16.669596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:16.669598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:16.669608 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:16.669631 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:16.669642 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:16.669928 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:16.669959 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 10:49:16.670014 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:49:16.670306 32498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:16.671870 32509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96fa3df9d86748728542151c2eee7138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fa3df9d86748728542151c2eee7138" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43783 } } }
I20260927 10:49:16.671850 32510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96fa3df9d86748728542151c2eee7138. Latest consensus state: current_term: 1 leader_uuid: "96fa3df9d86748728542151c2eee7138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96fa3df9d86748728542151c2eee7138" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 43783 } } }
I20260927 10:49:16.671991 32509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:16.671998 32510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138 [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:16.674783 32523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:16.674890 32523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:16.674983 32522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:16.675865 32522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:16.681087 32522 catalog_manager.cc:1347] Generated new cluster ID: 411a4018d110418d8a085049b90c0134
I20260927 10:49:16.681147 32522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:16.696363 32522 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:16.696743 32522 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:16.709254 32522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96fa3df9d86748728542151c2eee7138: Generated new TSK 0
I20260927 10:49:16.710034 32522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:16.716748 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32430
2026-09-27T10:49:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:16 dist-test-slave-wmg0 krb5kdc[32529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:16 dist-test-slave-wmg0 krb5kdc[32529](info): set up 2 sockets
Sep 27 10:49:16 dist-test-slave-wmg0 krb5kdc[32529](info): commencing operation
krb5kdc: starting...
W20260927 10:49:17.749881 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:17 dist-test-slave-wmg0 krb5kdc[32529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:17.874455 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33727
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:33727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:17.991678 32542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:17.991987 32542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:17.992069 32542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:17.996224 32542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:17.996318 32542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:17.996371 32542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:17.996408 32542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:17.996444 32542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:18.001842 32542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:33727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:18.003301 32542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:18.004596 32542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:18.013314 32547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:18.013314 32550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:18.013623 32542 server_base.cc:1034] running on GCE node
W20260927 10:49:18.013334 32548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:18.014249 32542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:18.015376 32542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:18.016534 32542 hybrid_clock.cc:648] HybridClock initialized: now 1790506158016513 us; error 33 us; skew 500 ppm
Sep 27 10:49:18 dist-test-slave-wmg0 krb5kdc[32529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506158, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:18.019558 32542 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:18.020437 32542 webserver.cc:460] Webserver started at http://127.10.96.254:41965/ using document root <none> and password file <none>
I20260927 10:49:18.021003 32542 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:18.021064 32542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:18.021281 32542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:18.023398 32542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "9920ec6c07f046a080faf45bcc9a142d"
format_stamp: "Formatted at 2026-09-27 10:49:18 on dist-test-slave-wmg0"
server_key: "dfd352e9a6cb3c490622ff1a03afa593"
server_key_iv: "1baa0ff8a7cba92ad258d17ccbc87c9e"
server_key_version: "encryptionkey@0"
I20260927 10:49:18.023974 32542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "9920ec6c07f046a080faf45bcc9a142d"
format_stamp: "Formatted at 2026-09-27 10:49:18 on dist-test-slave-wmg0"
server_key: "dfd352e9a6cb3c490622ff1a03afa593"
server_key_iv: "1baa0ff8a7cba92ad258d17ccbc87c9e"
server_key_version: "encryptionkey@0"
I20260927 10:49:18.027866 32542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 10:49:18.030354 32557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:18.031697 32542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 10:49:18.031845 32542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "9920ec6c07f046a080faf45bcc9a142d"
format_stamp: "Formatted at 2026-09-27 10:49:18 on dist-test-slave-wmg0"
server_key: "dfd352e9a6cb3c490622ff1a03afa593"
server_key_iv: "1baa0ff8a7cba92ad258d17ccbc87c9e"
server_key_version: "encryptionkey@0"
I20260927 10:49:18.031966 32542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:18.049013 32542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:18.052071 32542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:18.052284 32542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:18.059542 32542 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:33727
I20260927 10:49:18.059597 32609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:33727 every 8 connection(s)
I20260927 10:49:18.060562 32542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:18.061271 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32542
I20260927 10:49:18.061420 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:18.061718 10627 external_mini_cluster.cc:1428] Setting key f5f978c38ce116632c08d53029858fb9
I20260927 10:49:18.063915 32610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:18.069890 32610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d: Bootstrap starting.
Sep 27 10:49:18 dist-test-slave-wmg0 krb5kdc[32529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:18.072326 32610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:18.073081 32610 log.cc:824] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:18.074944 32610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d: No bootstrap required, opened a new log
I20260927 10:49:18.076256 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:18.063113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56935 (local address 127.10.96.254:33727)
0927 10:49:18.063561 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:18.063577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:18.063613 (+ 36us) server_negotiation.cc:407] Connection header received
0927 10:49:18.064333 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:18.064351 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:18.064694 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:18.065069 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:18.065916 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:18.066915 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:18.067777 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:18.068249 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:18.068440 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:18.071306 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:18.071311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:18.071324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:18.071351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:18.073286 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:18.073800 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:18.073807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:18.073820 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:18.073938 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:18.074257 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:18.074266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:18.074268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:18.074673 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:18.074890 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:18.075203 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 10:49:18.075514 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":143,"threads_started":1}
I20260927 10:49:18.078976 32610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9920ec6c07f046a080faf45bcc9a142d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33727 } }
I20260927 10:49:18.079253 32610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:18.079299 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:18.079346 32610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9920ec6c07f046a080faf45bcc9a142d, State: Initialized, Role: FOLLOWER
I20260927 10:49:18.079784 32610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9920ec6c07f046a080faf45bcc9a142d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33727 } }
I20260927 10:49:18.079937 32610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:18.080024 32610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:18.080118 32610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:18.081183 32610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9920ec6c07f046a080faf45bcc9a142d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33727 } }
I20260927 10:49:18.081573 32610 leader_election.cc:302] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9920ec6c07f046a080faf45bcc9a142d; no voters:
I20260927 10:49:18.081904 32610 leader_election.cc:288] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:18.081969 32620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:18.082149 32620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [term 1 LEADER]: Becoming Leader. State: Replica: 9920ec6c07f046a080faf45bcc9a142d, State: Running, Role: LEADER
I20260927 10:49:18.083292 32610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:18.083432 32620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9920ec6c07f046a080faf45bcc9a142d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33727 } }
I20260927 10:49:18.084638 32620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9920ec6c07f046a080faf45bcc9a142d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9920ec6c07f046a080faf45bcc9a142d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33727 } } }
I20260927 10:49:18.084764 32620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:18.085041 32621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9920ec6c07f046a080faf45bcc9a142d. Latest consensus state: current_term: 1 leader_uuid: "9920ec6c07f046a080faf45bcc9a142d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9920ec6c07f046a080faf45bcc9a142d" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 33727 } } }
I20260927 10:49:18.085141 32621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d [sys.catalog]: This master's current role is: LEADER
W20260927 10:49:18.088312 32635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:18.088407 32635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:18.088490 32631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:18.089322 32631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:18.090580 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:18.081484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47903 (local address 127.10.96.254:33727)
0927 10:49:18.081587 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:18.081590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:18.081783 (+ 193us) server_negotiation.cc:407] Connection header received
0927 10:49:18.081938 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:18.081942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:18.081988 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:18.082069 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:18.082867 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:18.083546 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:18.084571 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:18.084945 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:18.085034 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:18.087007 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:18.087011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:18.087013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:18.087037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:18.088776 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:18.089256 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:18.089260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:18.089262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:18.089349 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:18.089695 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:18.089698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:18.089700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:18.089925 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:18.090059 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:18.090322 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 10:49:18.090437 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 10:49:18.094759 32631 catalog_manager.cc:1347] Generated new cluster ID: 7604a1a5ac82443692869e0594369a54
I20260927 10:49:18.094828 32631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:18.123476 32631 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:18.124459 32631 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:18.132120 32631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9920ec6c07f046a080faf45bcc9a142d: Generated new TSK 0
I20260927 10:49:18.132843 32631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:18.142405 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32542
2026-09-27T10:49:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:18 dist-test-slave-wmg0 krb5kdc[32641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:18 dist-test-slave-wmg0 krb5kdc[32641](info): set up 2 sockets
Sep 27 10:49:18 dist-test-slave-wmg0 krb5kdc[32641](info): commencing operation
krb5kdc: starting...
W20260927 10:49:19.173710 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:19 dist-test-slave-wmg0 krb5kdc[32641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:19.295769 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37675
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:35693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:19.405450 32654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:19.405783 32654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:19.405864 32654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:19.409976 32654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:19.410061 32654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:19.410097 32654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:19.410153 32654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:19.410189 32654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:19.415660 32654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:35693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:19.416742 32654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:19.417851 32654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:19.426367 32660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:19.426381 32662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:19.426381 32659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:19.426633 32654 server_base.cc:1034] running on GCE node
I20260927 10:49:19.427258 32654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:19.428362 32654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:19.429557 32654 hybrid_clock.cc:648] HybridClock initialized: now 1790506159429547 us; error 29 us; skew 500 ppm
Sep 27 10:49:19 dist-test-slave-wmg0 krb5kdc[32641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506159, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:19.435951 32654 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:19.436908 32654 webserver.cc:460] Webserver started at http://127.10.96.254:33917/ using document root <none> and password file <none>
I20260927 10:49:19.437551 32654 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:19.437644 32654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:19.437879 32654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:19.439953 32654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "56cca9cc15524b0c9f774fea36eebf43"
format_stamp: "Formatted at 2026-09-27 10:49:19 on dist-test-slave-wmg0"
server_key: "c3fa1e11798228af771b7f0c5a1a7b2b"
server_key_iv: "ff347e5aec0cc4fda452a10f4ced5fc8"
server_key_version: "encryptionkey@0"
I20260927 10:49:19.440537 32654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "56cca9cc15524b0c9f774fea36eebf43"
format_stamp: "Formatted at 2026-09-27 10:49:19 on dist-test-slave-wmg0"
server_key: "c3fa1e11798228af771b7f0c5a1a7b2b"
server_key_iv: "ff347e5aec0cc4fda452a10f4ced5fc8"
server_key_version: "encryptionkey@0"
I20260927 10:49:19.444237 32654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:19.446636 32669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:19.447831 32654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:19.447976 32654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "56cca9cc15524b0c9f774fea36eebf43"
format_stamp: "Formatted at 2026-09-27 10:49:19 on dist-test-slave-wmg0"
server_key: "c3fa1e11798228af771b7f0c5a1a7b2b"
server_key_iv: "ff347e5aec0cc4fda452a10f4ced5fc8"
server_key_version: "encryptionkey@0"
I20260927 10:49:19.448108 32654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:19.471388 32654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:19.474972 32654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:19.475196 32654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:19.482146 32654 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:37675
I20260927 10:49:19.482182 32721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:37675 every 8 connection(s)
I20260927 10:49:19.483110 32654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:19.485946 32722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:19.491364 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43: Bootstrap starting.
I20260927 10:49:19.492257 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32654
I20260927 10:49:19.492432 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:19.492712 10627 external_mini_cluster.cc:1428] Setting key e9d0343b53a802855d31552670305101
I20260927 10:49:19.494052 32722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:19.494786 32722 log.cc:824] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:19.496768 32722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43: No bootstrap required, opened a new log
I20260927 10:49:19.500135 32722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cca9cc15524b0c9f774fea36eebf43" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37675 } }
I20260927 10:49:19.500377 32722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:19.500433 32722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cca9cc15524b0c9f774fea36eebf43, State: Initialized, Role: FOLLOWER
I20260927 10:49:19.500880 32722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cca9cc15524b0c9f774fea36eebf43" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37675 } }
I20260927 10:49:19.501036 32722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:19.501125 32722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:19.501256 32722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:19.501782 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:19.493810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60845 (local address 127.10.96.254:37675)
0927 10:49:19.494316 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:19.494331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:19.494366 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:19.495048 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:19.495067 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:19.495390 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:19.495762 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:19.496174 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:19.497202 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:19.498213 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:19.498697 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:19.498874 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:19.499729 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:19.499734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:19.499746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:19.500130 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:19.500181 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:19.500203 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:19.500318 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:19.500433 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 10:49:19.500809 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":154,"threads_started":1}
I20260927 10:49:19.502259 32722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cca9cc15524b0c9f774fea36eebf43" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37675 } }
I20260927 10:49:19.502597 32722 leader_election.cc:302] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56cca9cc15524b0c9f774fea36eebf43; no voters:
I20260927 10:49:19.502928 32722 leader_election.cc:288] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:19.503008 32727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:19.503185 32727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [term 1 LEADER]: Becoming Leader. State: Replica: 56cca9cc15524b0c9f774fea36eebf43, State: Running, Role: LEADER
I20260927 10:49:19.503479 32727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cca9cc15524b0c9f774fea36eebf43" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37675 } }
I20260927 10:49:19.504184 32722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:19.504659 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56cca9cc15524b0c9f774fea36eebf43. Latest consensus state: current_term: 1 leader_uuid: "56cca9cc15524b0c9f774fea36eebf43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cca9cc15524b0c9f774fea36eebf43" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37675 } } }
I20260927 10:49:19.504765 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:19.504866 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:19.505429 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56cca9cc15524b0c9f774fea36eebf43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cca9cc15524b0c9f774fea36eebf43" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 37675 } } }
I20260927 10:49:19.505537 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:19.505815 32732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:19.509054 32732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:19.513780 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:19.507215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57679 (local address 127.10.96.254:37675)
0927 10:49:19.507315 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:19.507324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:19.509766 (+ 2442us) server_negotiation.cc:407] Connection header received
0927 10:49:19.509913 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:19.509916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:19.509959 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:19.510055 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:19.510322 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:19.510839 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:19.511893 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:19.512229 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:19.512304 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:19.513274 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:19.513278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:19.513281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:19.513294 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:19.513322 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:19.513336 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:19.513385 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:19.513592 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 10:49:19.513647 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 10:49:19.515530 32732 catalog_manager.cc:1347] Generated new cluster ID: eb78e5dd73384dd68d5805085aa15e40
I20260927 10:49:19.515589 32732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:19.522984 32732 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:19.523831 32732 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:19.540248 32732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56cca9cc15524b0c9f774fea36eebf43: Generated new TSK 0
I20260927 10:49:19.541013 32732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:19.549698 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32654
2026-09-27T10:49:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:19 dist-test-slave-wmg0 krb5kdc[32753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:19 dist-test-slave-wmg0 krb5kdc[32753](info): set up 2 sockets
Sep 27 10:49:19 dist-test-slave-wmg0 krb5kdc[32753](info): commencing operation
krb5kdc: starting...
W20260927 10:49:20.589811 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:20 dist-test-slave-wmg0 krb5kdc[32753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:20.712330 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:34831
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:34831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:20.822336 32766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:20.822609 32766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:20.822691 32766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:20.826645 32766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:20.826695 32766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:20.826714 32766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:20.826774 32766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:20.826803 32766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:20.832096 32766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:34831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:34831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:20.833173 32766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:20.834228 32766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:20.842912 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:20.842927 306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:20.842921 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:20.842957 32766 server_base.cc:1034] running on GCE node
I20260927 10:49:20.843767 32766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:20.844947 32766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:20.846122 32766 hybrid_clock.cc:648] HybridClock initialized: now 1790506160846106 us; error 30 us; skew 500 ppm
Sep 27 10:49:20 dist-test-slave-wmg0 krb5kdc[32753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:20.849004 32766 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:20.849915 32766 webserver.cc:460] Webserver started at http://127.10.96.254:40917/ using document root <none> and password file <none>
I20260927 10:49:20.850437 32766 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:20.850494 32766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:20.850659 32766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:20.852698 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "ae63ff863bd44223a6010943f2d5b4fb"
format_stamp: "Formatted at 2026-09-27 10:49:20 on dist-test-slave-wmg0"
server_key: "f3280bbf732e7b4bc238c09e10b39017"
server_key_iv: "d1081567779bee092e3b44d37648364e"
server_key_version: "encryptionkey@0"
I20260927 10:49:20.853226 32766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "ae63ff863bd44223a6010943f2d5b4fb"
format_stamp: "Formatted at 2026-09-27 10:49:20 on dist-test-slave-wmg0"
server_key: "f3280bbf732e7b4bc238c09e10b39017"
server_key_iv: "d1081567779bee092e3b44d37648364e"
server_key_version: "encryptionkey@0"
I20260927 10:49:20.856910 32766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:20.859431 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:20.860756 32766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 10:49:20.860901 32766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "ae63ff863bd44223a6010943f2d5b4fb"
format_stamp: "Formatted at 2026-09-27 10:49:20 on dist-test-slave-wmg0"
server_key: "f3280bbf732e7b4bc238c09e10b39017"
server_key_iv: "d1081567779bee092e3b44d37648364e"
server_key_version: "encryptionkey@0"
I20260927 10:49:20.861018 32766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:20.865777 32766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:20.868999 32766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:20.869222 32766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:20.876518 32766 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:34831
I20260927 10:49:20.876540 365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:34831 every 8 connection(s)
I20260927 10:49:20.877554 32766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:20.878657 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 32766
I20260927 10:49:20.878816 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:20.879117 10627 external_mini_cluster.cc:1428] Setting key d902219559045161e812eab43a99ba3d
I20260927 10:49:20.880861 366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:20.884392 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:20.880586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38889 (local address 127.10.96.254:34831)
0927 10:49:20.881081 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:20.881096 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:20.881136 (+ 40us) server_negotiation.cc:407] Connection header received
0927 10:49:20.881858 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:20.881876 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:20.882154 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:20.882503 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:20.883269 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:20.883274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:20.883286 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:20.883296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:20.883347 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:20.883364 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:20.883521 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:20.883589 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 10:49:20.883740 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":149,"threads_started":1}
I20260927 10:49:20.886989 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb: Bootstrap starting.
I20260927 10:49:20.887249 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:20.889694 366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:20.890625 366 log.cc:824] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:20.891330 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:20.889545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35447 (local address 127.10.96.254:34831)
0927 10:49:20.889661 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:20.889666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:20.889908 (+ 242us) server_negotiation.cc:407] Connection header received
0927 10:49:20.890040 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:20.890042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:20.890072 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:20.890138 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:20.890904 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:20.890907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:20.890909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 10:49:20.890919 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:20.890968 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 10:49:20.890978 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:20.891077 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:20.891167 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 10:49:20.891206 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:49:20.892750 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb: No bootstrap required, opened a new log
I20260927 10:49:20.896435 366 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae63ff863bd44223a6010943f2d5b4fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34831 } }
I20260927 10:49:20.896651 366 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 10:49:20.896687 366 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae63ff863bd44223a6010943f2d5b4fb, State: Initialized, Role: FOLLOWER
I20260927 10:49:20.897086 366 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae63ff863bd44223a6010943f2d5b4fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34831 } }
I20260927 10:49:20.897207 366 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 10:49:20.897253 366 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 10:49:20.897316 366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:20.898205 366 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae63ff863bd44223a6010943f2d5b4fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34831 } }
I20260927 10:49:20.898531 366 leader_election.cc:302] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae63ff863bd44223a6010943f2d5b4fb; no voters:
I20260927 10:49:20.898783 366 leader_election.cc:288] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 10:49:20.898910 376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:20.899174 376 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [term 1 LEADER]: Becoming Leader. State: Replica: ae63ff863bd44223a6010943f2d5b4fb, State: Running, Role: LEADER
I20260927 10:49:20.899492 376 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae63ff863bd44223a6010943f2d5b4fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34831 } }
I20260927 10:49:20.900111 366 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:20.901177 377 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae63ff863bd44223a6010943f2d5b4fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae63ff863bd44223a6010943f2d5b4fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34831 } } }
I20260927 10:49:20.901309 377 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:20.901662 378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae63ff863bd44223a6010943f2d5b4fb. Latest consensus state: current_term: 1 leader_uuid: "ae63ff863bd44223a6010943f2d5b4fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae63ff863bd44223a6010943f2d5b4fb" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 34831 } } }
I20260927 10:49:20.901727 385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:20.901831 378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:20.905103 385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:20.911105 385 catalog_manager.cc:1347] Generated new cluster ID: 540e341df06c40a58ae0595b2a4a44ad
I20260927 10:49:20.911183 385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:20.936134 385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:20.936506 385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:20.947937 385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae63ff863bd44223a6010943f2d5b4fb: Generated new TSK 0
I20260927 10:49:20.948872 385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 10:49:20.962401 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 32766
2026-09-27T10:49:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1412 ms)
[----------] 18 tests from AuthTokenIssuingTest (25338 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:21 dist-test-slave-wmg0 krb5kdc[397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:21 dist-test-slave-wmg0 krb5kdc[397](info): set up 2 sockets
Sep 27 10:49:21 dist-test-slave-wmg0 krb5kdc[397](info): commencing operation
krb5kdc: starting...
W20260927 10:49:22.005959 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:22.144320 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35271
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:35271,127.10.96.253:43443,127.10.96.252:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:22.256826 410 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:49:22.257138 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:22.257164 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:22.257256 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:22.261377 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:22.261441 410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:22.261487 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:22.261551 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:22.261582 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:22.267138 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:35271,127.10.96.253:43443,127.10.96.252:34485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:22.268278 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:22.269340 410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:22.278136 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:22.278115 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:22.278198 410 server_base.cc:1034] running on GCE node
W20260927 10:49:22.278115 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:22.278985 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:22.280162 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:22.281325 410 hybrid_clock.cc:648] HybridClock initialized: now 1790506162281305 us; error 32 us; skew 500 ppm
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:22.284229 410 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:22.285113 410 webserver.cc:460] Webserver started at http://127.10.96.254:36683/ using document root <none> and password file <none>
I20260927 10:49:22.285715 410 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:22.285779 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:22.285964 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:22.288004 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "5961f6f10c1d469cb1bc5a3687402d67"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "50da42a7236bef65bfe51909f3574a88"
server_key_iv: "0c79cda91815a072ecc7b9e62e270497"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.288533 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "5961f6f10c1d469cb1bc5a3687402d67"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "50da42a7236bef65bfe51909f3574a88"
server_key_iv: "0c79cda91815a072ecc7b9e62e270497"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.292344 410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:49:22.295091 425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:22.296454 410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:22.296612 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "5961f6f10c1d469cb1bc5a3687402d67"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "50da42a7236bef65bfe51909f3574a88"
server_key_iv: "0c79cda91815a072ecc7b9e62e270497"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.296723 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:22.312660 410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:22.315677 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:22.315888 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:22.322811 410 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:35271
I20260927 10:49:22.323112 477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:35271 every 8 connection(s)
I20260927 10:49:22.323789 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:22.326606 478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:22.330268 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.330837 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 410
I20260927 10:49:22.331001 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:22.331264 10627 external_mini_cluster.cc:1428] Setting key 7af0688d0941c54f95cf3323d97d60a2
WARNING: no policy specified for kudu/127.10.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.253@KRBTEST.COM" created.
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:22.348773 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.333701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.10.96.254:35271)
0927 10:49:22.333959 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.333968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.334003 (+ 35us) server_negotiation.cc:407] Connection header received
0927 10:49:22.335308 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.335313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.335502 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.335681 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.336832 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.337765 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.339278 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.339658 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.339771 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.343898 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.343904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.343914 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.343941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.345659 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.346120 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.346125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.346127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.346230 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.346540 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.346543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.346545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.346910 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.347047 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.347426 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.347695 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260927 10:49:22.348773 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.332814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35271 (local address 127.0.0.1:60616)
0927 10:49:22.333451 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.333542 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.334972 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.335720 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.335728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.336253 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.336652 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.336676 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.337892 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.337899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.339159 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.339170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.339822 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.339830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.339940 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.340558 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.340574 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.343748 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.345768 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.345776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.345786 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.346012 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.346324 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.346329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.346331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.346446 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.347031 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.347037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.347174 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.347340 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.347743 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":110,"thread_start_us":150,"threads_started":1}
I20260927 10:49:22.351680 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.352542 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.352240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49230)
0927 10:49:22.352384 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.352474 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 10:49:22.352733 426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
I20260927 10:49:22.353603 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.353363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49232)
0927 10:49:22.353498 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.353548 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 10:49:22.353767 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:43443: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab.
Entry for principal kudu/127.10.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab.
I20260927 10:49:22.360086 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:43443
--webserver_interface=127.10.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab
--principal=kudu/127.10.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:35271,127.10.96.253:43443,127.10.96.252:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:49:22.402983 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } attempt: 1
I20260927 10:49:22.403846 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.403555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49248)
0927 10:49:22.403703 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.403765 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 10:49:22.404664 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.404427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49256)
0927 10:49:22.404537 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.404593 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 10:49:22.404804 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:43443: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
W20260927 10:49:22.471915 485 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:49:22.472198 485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:22.472220 485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:22.472259 485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:49:22.472987 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } attempt: 2
I20260927 10:49:22.473887 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.473572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49266)
0927 10:49:22.473729 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.473799 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 10:49:22.474654 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.474439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49268)
0927 10:49:22.474553 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.474600 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:49:22.474799 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:43443: Network error: Client connection negotiation failed: client connection to 127.10.96.253:43443: connect: Connection refused (error 111)
W20260927 10:49:22.477038 485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:22.477093 485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:22.477113 485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:22.477129 485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:22.477193 485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:22.482774 485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:35271,127.10.96.253:43443,127.10.96.252:34485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab
--principal=kudu/127.10.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:22.483848 485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:22.484918 485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:22.493660 491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:22.493660 490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:22.493661 493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:22.493927 485 server_base.cc:1034] running on GCE node
I20260927 10:49:22.494683 485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:22.495839 485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:22.497023 485 hybrid_clock.cc:648] HybridClock initialized: now 1790506162497008 us; error 29 us; skew 500 ppm
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:22.500115 485 init.cc:377] Logged in from keytab as kudu/127.10.96.253@KRBTEST.COM (short username kudu)
I20260927 10:49:22.501133 485 webserver.cc:460] Webserver started at http://127.10.96.253:37503/ using document root <none> and password file <none>
I20260927 10:49:22.501755 485 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:22.501819 485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:22.502032 485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:22.504053 485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data/instance:
uuid: "b9068139b7004eb9ba286f82ccd73641"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "c5da4a5e9f3e79a9730c45ad19116493"
server_key_iv: "bd20f3c49e356fcf6fc3251546edfb7a"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.504626 485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal/instance:
uuid: "b9068139b7004eb9ba286f82ccd73641"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "c5da4a5e9f3e79a9730c45ad19116493"
server_key_iv: "bd20f3c49e356fcf6fc3251546edfb7a"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.508335 485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 10:49:22.510777 500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:22.512037 485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:22.512178 485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal
uuid: "b9068139b7004eb9ba286f82ccd73641"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "c5da4a5e9f3e79a9730c45ad19116493"
server_key_iv: "bd20f3c49e356fcf6fc3251546edfb7a"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.512295 485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:22.535085 485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:22.538833 485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:22.539059 485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:22.546195 485 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.253:43443
I20260927 10:49:22.546530 552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.253:43443 every 8 connection(s)
I20260927 10:49:22.547215 485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
I20260927 10:49:22.550021 553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:22.553795 553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.556648 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 485
I20260927 10:49:22.556782 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-1/wal/instance
I20260927 10:49:22.557054 10627 external_mini_cluster.cc:1428] Setting key eff06074b514538359266f87333b4eb9
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.96.252@KRBTEST.COM; defaulting to no policy
I20260927 10:49:22.565980 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.252@KRBTEST.COM" created.
I20260927 10:49:22.567776 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.555691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.10.96.254:35271)
0927 10:49:22.555844 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.555848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.556679 (+ 831us) server_negotiation.cc:407] Connection header received
0927 10:49:22.557870 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.557873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.557940 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.558049 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.559239 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.559767 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.560892 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.561134 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.561250 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.563773 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.563776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.563778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.563802 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.565335 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.565919 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.565953 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.565955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.566031 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.566446 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.566454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.566456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.566713 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.566833 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.567490 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.567609 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:49:22.568648 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.556060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35271 (local address 127.0.0.1:60622)
0927 10:49:22.556522 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.556565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.557650 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.558225 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.558232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.558592 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.559083 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.559097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.559900 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.559908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.560763 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.560772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.561288 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.561292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.561520 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.562189 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.562205 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.563597 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.565519 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.565526 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.565544 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.565811 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.566126 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.566130 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.566133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.566319 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.566887 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.566893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.567220 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.567412 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.567743 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260927 10:49:22.570194 553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:49:22.578543 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.566494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49276)
0927 10:49:22.566659 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.566672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.566764 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.567877 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.567884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.568138 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.568283 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.568294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.569893 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.569899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.571438 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.571452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.572172 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.572176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.572292 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.572815 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.572829 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.574142 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.576636 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.576639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.576641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.577007 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.577371 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.577374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.577378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.577504 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.577967 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.577971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.578098 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.578245 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.578399 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 10:49:22.579043 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.566605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.10.96.253:43443)
0927 10:49:22.567385 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.567397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.567423 (+ 26us) server_negotiation.cc:407] Connection header received
0927 10:49:22.567473 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.567483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.567707 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.568450 (+ 743us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.568591 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.569764 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.571565 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.572054 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.572168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.574299 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.574311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.574326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.574354 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.576499 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.577142 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.577152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.577154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.577268 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.577597 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.577599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.577600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.577854 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.577961 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.578361 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.578789 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":106,"threads_started":1}
I20260927 10:49:22.581099 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.581290 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.570795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.10.96.253:43443)
0927 10:49:22.571137 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.571148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.571167 (+ 19us) server_negotiation.cc:407] Connection header received
0927 10:49:22.571277 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.571283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.571317 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.571417 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.571829 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.572708 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.573954 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.574334 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.574434 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.576858 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.576862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.576864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.576890 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.579368 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.579801 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.579804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.579805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.579898 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.580175 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.580177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.580179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.580398 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.580492 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.580830 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.580981 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":291,"thread_start_us":178,"threads_started":1}
I20260927 10:49:22.581507 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.570757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49292)
0927 10:49:22.571113 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.571128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.571206 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.571425 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.571428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.571612 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.571699 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.571705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.572844 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.572850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.573805 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.573818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.574550 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.574554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.574664 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.575126 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.575140 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.576697 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.579472 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.579475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.579477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.579708 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.579992 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.579994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.579996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.580075 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.580488 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.580490 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.580644 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.580775 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.580909 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260927 10:49:22.581900 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.581592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47664)
0927 10:49:22.581769 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.581829 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 10:49:22.582296 553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.582630 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.582368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47678)
0927 10:49:22.582509 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.582554 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 10:49:22.582814 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34485: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
I20260927 10:49:22.583016 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.582708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47694)
0927 10:49:22.582857 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.582951 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 10:49:22.583230 502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab.
Entry for principal kudu/127.10.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab.
I20260927 10:49:22.584033 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.583816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47708)
0927 10:49:22.583922 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.583968 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 10:49:22.584177 553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34485: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
I20260927 10:49:22.584805 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:34485
--webserver_interface=127.10.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab
--principal=kudu/127.10.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:35271,127.10.96.253:43443,127.10.96.252:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:49:22.614013 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } attempt: 1
I20260927 10:49:22.614888 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.614547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47714)
0927 10:49:22.614721 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.614794 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:49:22.615721 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.615462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47718)
0927 10:49:22.615582 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.615634 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:49:22.615875 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34485: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
I20260927 10:49:22.633433 553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } attempt: 1
I20260927 10:49:22.634459 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.634098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47724)
0927 10:49:22.634291 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.634377 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 10:49:22.635269 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.635035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47728)
0927 10:49:22.635149 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.635206 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 10:49:22.635411 553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34485: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
I20260927 10:49:22.691044 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } attempt: 2
I20260927 10:49:22.691926 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.691580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47732)
0927 10:49:22.691751 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.691818 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 10:49:22.692741 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.692494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47736)
0927 10:49:22.692632 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.692680 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 10:49:22.692943 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34485: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
W20260927 10:49:22.700531 561 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:49:22.700865 561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:22.700929 561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:22.701005 561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:49:22.703580 553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } attempt: 2
I20260927 10:49:22.704334 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.704045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47740)
0927 10:49:22.704183 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.704255 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 10:49:22.705811 561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:22.705885 561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:22.705926 561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:22.705967 561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:22.706002 561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:22.706173 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.705058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47750)
0927 10:49:22.706050 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.706109 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":841}
W20260927 10:49:22.706296 553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34485: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34485: connect: Connection refused (error 111)
I20260927 10:49:22.711634 561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:35271,127.10.96.253:43443,127.10.96.252:34485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab
--principal=kudu/127.10.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:34485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:22.712808 561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:22.713959 561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:22.722549 569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:22.722576 566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:22.722599 567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:22.722838 561 server_base.cc:1034] running on GCE node
I20260927 10:49:22.723416 561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:22.724489 561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:22.725658 561 hybrid_clock.cc:648] HybridClock initialized: now 1790506162725645 us; error 25 us; skew 500 ppm
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:22.728329 561 init.cc:377] Logged in from keytab as kudu/127.10.96.252@KRBTEST.COM (short username kudu)
I20260927 10:49:22.729256 561 webserver.cc:460] Webserver started at http://127.10.96.252:41073/ using document root <none> and password file <none>
I20260927 10:49:22.729876 561 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:22.729966 561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:22.730197 561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:22.732229 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data/instance:
uuid: "1830722bcec24bdc951ab28d08657e17"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "bc38282bed92f657f337e1ae9e38c008"
server_key_iv: "390a973bcce25511d3463f12ad073082"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.732817 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal/instance:
uuid: "1830722bcec24bdc951ab28d08657e17"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "bc38282bed92f657f337e1ae9e38c008"
server_key_iv: "390a973bcce25511d3463f12ad073082"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.736549 561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 10:49:22.739140 576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:22.740374 561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:22.740525 561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal
uuid: "1830722bcec24bdc951ab28d08657e17"
format_stamp: "Formatted at 2026-09-27 10:49:22 on dist-test-slave-wmg0"
server_key: "bc38282bed92f657f337e1ae9e38c008"
server_key_iv: "390a973bcce25511d3463f12ad073082"
server_key_version: "encryptionkey@0"
I20260927 10:49:22.740640 561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:22.777854 561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:22.781040 561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:22.781262 561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:22.788378 561 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.252:34485
I20260927 10:49:22.788676 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.252:34485 every 8 connection(s)
I20260927 10:49:22.789397 561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
I20260927 10:49:22.791579 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 561
I20260927 10:49:22.791704 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790505884435945-10627-0/minicluster-data/master-2/wal/instance
I20260927 10:49:22.792018 10627 external_mini_cluster.cc:1428] Setting key 96120201c7b8dc7dd91dcb84b412ea22
I20260927 10:49:22.792164 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } attempt: 3
I20260927 10:49:22.792666 629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:22.797482 553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } attempt: 3
I20260927 10:49:22.798043 629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
I20260927 10:49:22.807834 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.797975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47782)
0927 10:49:22.798179 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.798194 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.798276 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.798699 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.798702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.798899 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.799018 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.799025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.800115 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.800119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.800961 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.800974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.801714 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.801720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.801805 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.802449 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.802463 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.803878 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.806225 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.806231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.806236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.806496 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.806814 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.806817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.806819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.806904 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.807324 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.807328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.807427 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.807514 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.807659 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:22.808501 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.792694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47768)
0927 10:49:22.792911 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.792928 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.793020 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.795578 (+ 2558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.795581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.795819 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.795960 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.795969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.798406 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.798410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.799167 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.799181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.800009 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.800013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.800122 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.800595 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.800609 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.802267 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.806826 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.806832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.806836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.807094 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.807426 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.807429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.807431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.807527 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.807982 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.807988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.808105 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.808203 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.808337 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 10:49:22.809679 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.793690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.10.96.252:34485)
0927 10:49:22.794103 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.794123 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.794170 (+ 47us) server_negotiation.cc:407] Connection header received
0927 10:49:22.794879 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.794897 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.795240 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.795589 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.796408 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.797313 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.798519 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.799229 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.799376 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.803094 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.803102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.803107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.803138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.805667 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.806171 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.806180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.806190 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.806352 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.806670 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.806675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.806677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.807210 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.807396 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.808149 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.808547 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":278,"thread_start_us":143,"threads_started":1}
I20260927 10:49:22.809834 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.794090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.10.96.252:34485)
0927 10:49:22.794378 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.794387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.794402 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:49:22.794923 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.794929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.795242 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.796084 (+ 842us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.796205 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.797365 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.799316 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.799868 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.800339 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.802454 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.802462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.802534 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.802565 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.806705 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.807217 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.807221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.807225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.807311 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.807637 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.807642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.807647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.807853 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.808787 (+ 934us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.808861 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.809008 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":293,"server-negotiator.queue_time_us":220,"spinlock_wait_cycles":15104,"thread_start_us":85,"threads_started":1}
I20260927 10:49:22.809664 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.798063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.10.96.252:34485)
0927 10:49:22.798369 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.798377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.798391 (+ 14us) server_negotiation.cc:407] Connection header received
0927 10:49:22.798426 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.798429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.798555 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.799129 (+ 574us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.799215 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.800002 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.801180 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.801578 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.801681 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.804034 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.804038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.804041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.804067 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.806079 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.806598 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.806600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.806601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.806706 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.807028 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.807030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.807032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.807203 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.808139 (+ 936us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.808202 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.808593 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":166,"server-negotiator.queue_time_us":250,"thread_start_us":150,"threads_started":1}
I20260927 10:49:22.813005 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.800979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.10.96.254:35271)
0927 10:49:22.801128 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.801134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.801551 (+ 417us) server_negotiation.cc:407] Connection header received
0927 10:49:22.801756 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.801759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.801803 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.801852 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.802827 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.803667 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.804968 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.805410 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.805573 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.809232 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.809236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.809238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.809261 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.810960 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.811488 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.811492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.811493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.811679 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.811986 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.811989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.811990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.812251 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.812331 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.812760 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.812876 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:49:22.813730 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.800102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:35271 (local address 127.0.0.1:60636)
0927 10:49:22.801400 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.801474 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.801641 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.802054 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.802060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.802531 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.802692 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.802700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.803797 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.803801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.804843 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.804857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.805571 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.805575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.805690 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.806752 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.806770 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.809061 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.811088 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.811098 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.811103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.811375 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.811785 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.811788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.811789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.811878 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.812416 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.812435 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.812546 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.813358 (+ 812us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.813577 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":92,"threads_started":1}
I20260927 10:49:22.814267 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67: Bootstrap starting.
I20260927 10:49:22.814395 553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: Bootstrap starting.
I20260927 10:49:22.815083 629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.817239 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:22.817879 553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:22.818279 478 log.cc:824] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:22.818678 553 log.cc:824] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: Log is configured to *not* fsync() on all Append() calls
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:22.820904 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67: No bootstrap required, opened a new log
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:49:22.823662 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.813583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.10.96.254:35271)
0927 10:49:22.813682 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.813686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.813698 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:49:22.813730 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.813732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.813769 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.813840 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.814663 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.815396 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.816585 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.816878 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.816969 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.819876 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.819880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.819882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.819906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.821698 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.822352 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.822355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.822356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.822440 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.822758 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.822762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.822765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.823004 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.823183 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.823389 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.823499 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 10:49:22.825219 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.815594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:43443 (local address 127.0.0.1:49308)
0927 10:49:22.815968 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.815982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.816055 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.816345 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.816348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.816524 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.816628 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.816634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.817707 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.817712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.818682 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.818695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.819484 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.819487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.819577 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.820031 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.820045 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.821752 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.823705 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.823709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.823711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.823947 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.824244 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.824247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.824249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.824330 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.824765 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.824769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.824860 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.824961 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.825095 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":106,"threads_started":1}
I20260927 10:49:22.825691 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.816032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.10.96.253:43443)
0927 10:49:22.816150 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.816154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.816166 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:49:22.816205 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.816207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.816241 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.816331 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.816759 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.817517 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.818868 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.819253 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.819334 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.821889 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.821892 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.821895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.821919 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.823579 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.824061 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.824064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.824066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.824144 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.824413 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.824417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.824419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.824656 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.824764 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.825426 (+ 662us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.825572 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 10:49:22.826309 629 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:22.826722 553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: No bootstrap required, opened a new log
I20260927 10:49:22.830822 553 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.831085 553 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9068139b7004eb9ba286f82ccd73641, State: Initialized, Role: FOLLOWER
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:49:22.831446 553 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.831947 639 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.832072 639 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 10:49:22 dist-test-slave-wmg0 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506162, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
I20260927 10:49:22.832913 553 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:22.833868 478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.834204 478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5961f6f10c1d469cb1bc5a3687402d67, State: Initialized, Role: FOLLOWER
I20260927 10:49:22.834606 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.836187 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.824996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.10.96.253:43443)
0927 10:49:22.825103 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.825109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.825127 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:49:22.825296 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.825301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.825366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.825443 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.826555 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.827275 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.828720 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.829082 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.829176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.832082 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.832086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.832088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.832114 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.834160 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.834673 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.834678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.834682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.834763 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.835074 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.835077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.835091 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.835325 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.835518 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.835903 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.836030 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 10:49:22.836289 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.827573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34485 (local address 127.0.0.1:47790)
0927 10:49:22.827700 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:22.827714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:22.827789 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:22.828049 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:22.828051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:22.828213 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:22.828271 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.828276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.829121 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.829125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.829927 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:22.829937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.830601 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.830605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:22.830681 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.831162 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:22.831176 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:22.832610 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:22.834856 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.834861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.834863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.835110 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.835369 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:22.835371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:22.835373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:22.835449 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:22.835851 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:22.835854 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:22.835940 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:49:22.836019 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 10:49:22.836134 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 10:49:22.836756 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.827671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.10.96.252:34485)
0927 10:49:22.827896 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.827898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.827906 (+ 8us) server_negotiation.cc:407] Connection header received
0927 10:49:22.827929 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.827932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.827961 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.828037 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.828376 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.829001 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.830054 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.830471 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.830596 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.832760 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.832763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.832766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.832791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.834752 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.835212 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.835214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.835216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.835280 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.835552 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.835558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.835559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.835739 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.836438 (+ 699us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.836510 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.836611 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260927 10:49:22.837671 478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:22.836591 485 master_runner.cc:186] Error getting master registration for 127.10.96.254:35271: OK, instance_id { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" instance_seqno: 1790506162320967 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:49:22.839855 629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: Bootstrap starting.
I20260927 10:49:22.840018 485 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:49:22.840021 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:22.841214 452 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5961f6f10c1d469cb1bc5a3687402d67"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60650
I20260927 10:49:22.841485 452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:49:22.841614 452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:22.842307 629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:22.842967 452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.843106 629 log.cc:824] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:22.843813 651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.844465 527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5961f6f10c1d469cb1bc5a3687402d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9068139b7004eb9ba286f82ccd73641"
I20260927 10:49:22.844434 651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:22.844697 527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [term 0 FOLLOWER]: Advancing to term 1
W20260927 10:49:22.845146 650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:22.845218 650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:22.843914 452 leader_election.cc:288] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [CANDIDATE]: Term 1 election: Requested vote from peers b9068139b7004eb9ba286f82ccd73641 (127.10.96.253:43443), 1830722bcec24bdc951ab28d08657e17 (127.10.96.252:34485)
I20260927 10:49:22.847757 527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5961f6f10c1d469cb1bc5a3687402d67 in term 1.
I20260927 10:49:22.848225 426 leader_election.cc:302] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5961f6f10c1d469cb1bc5a3687402d67, b9068139b7004eb9ba286f82ccd73641; no voters:
I20260927 10:49:22.848534 651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:22.848865 651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [term 1 LEADER]: Becoming Leader. State: Replica: 5961f6f10c1d469cb1bc5a3687402d67, State: Running, Role: LEADER
I20260927 10:49:22.849412 651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.852885 651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5961f6f10c1d469cb1bc5a3687402d67. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.853077 651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:22.851274 410 master_runner.cc:186] Error getting master registration for 127.10.96.252:34485: OK, instance_id { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" instance_seqno: 1790506162786446 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 10:49:22.855388 410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:49:22.856364 664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:22.856443 664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:22.857313 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.847212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.10.96.254:35271)
0927 10:49:22.847320 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.847325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.848398 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 10:49:22.848600 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.848603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.848649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.848698 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.849761 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.850501 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.851588 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.851912 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.851983 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.853892 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.853896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.853899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.853929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.855657 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.856076 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.856080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.856082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.856171 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.856466 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.856469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.856471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.856649 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.856806 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.857029 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.857141 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:49:22.864322 629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: No bootstrap required, opened a new log
I20260927 10:49:22.867903 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.847398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.10.96.252:34485)
0927 10:49:22.847508 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.847512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.857532 (+ 10020us) server_negotiation.cc:407] Connection header received
0927 10:49:22.857709 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.857716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.857776 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.857832 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.858724 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.859502 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.860587 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.860818 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.860944 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.862516 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.862519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.862522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.862547 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.865038 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.866551 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.866554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.866556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.866646 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.866957 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.866961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.866962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.867169 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.867277 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.867617 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.867740 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 10:49:22.869933 629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.870484 629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1830722bcec24bdc951ab28d08657e17, State: Initialized, Role: FOLLOWER
I20260927 10:49:22.871165 629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } }
I20260927 10:49:22.873704 603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5961f6f10c1d469cb1bc5a3687402d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1830722bcec24bdc951ab28d08657e17"
I20260927 10:49:22.873951 603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:22.875193 674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:22.875367 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:22.847830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.10.96.253:43443)
0927 10:49:22.847940 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.847945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.862253 (+ 14308us) server_negotiation.cc:407] Connection header received
0927 10:49:22.862454 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.862458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.862518 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.862571 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.863536 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.864267 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.869296 (+ 5029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:22.869676 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:22.869756 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:22.871669 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:22.871676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:22.871681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:22.871714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:22.873421 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.873932 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.873936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.873938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.874027 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:22.874344 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:22.874350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:22.874355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:22.874573 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:22.874672 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:22.875031 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 10:49:22.875159 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 10:49:22.875402 603 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5961f6f10c1d469cb1bc5a3687402d67 in term 1.
I20260927 10:49:22.876313 674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 10:49:22.876448 629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:22.881393 603 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [term 1 FOLLOWER]: Refusing update from remote peer 5961f6f10c1d469cb1bc5a3687402d67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:22.881603 527 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [term 1 FOLLOWER]: Refusing update from remote peer 5961f6f10c1d469cb1bc5a3687402d67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:22.882215 672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:22.885524 672 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:22.885743 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.885962 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:22.890233 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5961f6f10c1d469cb1bc5a3687402d67. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.890417 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:22.890482 639 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5961f6f10c1d469cb1bc5a3687402d67. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.890609 639 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:22.894163 651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.894280 651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:22.894379 639 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.894487 639 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:22.895733 684 mvcc.cc:205] Tried to move back new op lower bound from 7333913243155775488 to 7333913243037057024. Current Snapshot: MvccSnapshot[applied={T|T < 7333913243155775488}]
I20260927 10:49:22.896467 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.896580 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:22.897949 689 mvcc.cc:205] Tried to move back new op lower bound from 7333913243155775488 to 7333913243037057024. Current Snapshot: MvccSnapshot[applied={T|T < 7333913243155775488}]
I20260927 10:49:22.897996 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5961f6f10c1d469cb1bc5a3687402d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5961f6f10c1d469cb1bc5a3687402d67" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 35271 } } peers { permanent_uuid: "b9068139b7004eb9ba286f82ccd73641" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 43443 } } peers { permanent_uuid: "1830722bcec24bdc951ab28d08657e17" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34485 } } }
I20260927 10:49:22.898195 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:22.898334 690 mvcc.cc:205] Tried to move back new op lower bound from 7333913243155775488 to 7333913243037057024. Current Snapshot: MvccSnapshot[applied={T|T < 7333913243155775488}]
W20260927 10:49:22.901388 688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:22.901504 688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:22.909857 674 catalog_manager.cc:1347] Generated new cluster ID: cf9328f936cf464fa78866256ad57c4a
I20260927 10:49:22.909950 674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:22.935998 674 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:22.937336 674 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:22.950747 674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5961f6f10c1d469cb1bc5a3687402d67: Generated new TSK 0
I20260927 10:49:22.951682 674 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:49:22.978082 632 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:22.975631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.10.96.252:34485)
0927 10:49:22.975752 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.975757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.976941 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 10:49:22.977151 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.977155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.977207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.977261 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.977937 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47808: BlockingRecv error: recv got EOF from 127.0.0.1:47808 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 10:49:22.979446 482 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:22.975133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.10.96.254:35271)
0927 10:49:22.975272 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.975279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.978544 (+ 3265us) server_negotiation.cc:407] Connection header received
0927 10:49:22.978717 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.978721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.978779 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.978837 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.979324 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60670: BlockingRecv error: recv got EOF from 127.0.0.1:60670 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 10:49:22.980741 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:22.976349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.10.96.253:43443)
0927 10:49:22.976465 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.976469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.979897 (+ 3428us) server_negotiation.cc:407] Connection header received
0927 10:49:22.980071 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.980074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.980124 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.980173 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.980625 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49332: BlockingRecv error: recv got EOF from 127.0.0.1:49332 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:49:22.985443 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:22.983846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.10.96.253:43443)
0927 10:49:22.983963 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.983967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.984481 (+ 514us) server_negotiation.cc:407] Connection header received
0927 10:49:22.984698 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.984702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.984775 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.984828 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.985338 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49348: BlockingRecv error: recv got EOF from 127.0.0.1:49348 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 10:49:22.990510 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:22.989005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.10.96.253:43443)
0927 10:49:22.989103 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.989107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.989627 (+ 520us) server_negotiation.cc:407] Connection header received
0927 10:49:22.989826 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.989830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.989880 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.989941 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.990410 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49358: BlockingRecv error: recv got EOF from 127.0.0.1:49358 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 10:49:22.996042 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:22.994571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.10.96.253:43443)
0927 10:49:22.994663 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:22.994667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:22.995196 (+ 529us) server_negotiation.cc:407] Connection header received
0927 10:49:22.995364 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:22.995367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:22.995414 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:22.995463 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:22.995948 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49362: BlockingRecv error: recv got EOF from 127.0.0.1:49362 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 10:49:23.004287 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.002442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.10.96.253:43443)
0927 10:49:23.002593 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.002598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.003266 (+ 668us) server_negotiation.cc:407] Connection header received
0927 10:49:23.003478 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.003482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.003551 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.003606 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.004166 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49366: BlockingRecv error: recv got EOF from 127.0.0.1:49366 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 10:49:23.016819 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.014983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.10.96.253:43443)
0927 10:49:23.015101 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.015107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.015700 (+ 593us) server_negotiation.cc:407] Connection header received
0927 10:49:23.015952 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.015956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.016016 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.016069 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.016697 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49378: BlockingRecv error: recv got EOF from 127.0.0.1:49378 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 10:49:23.037354 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.035886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.10.96.253:43443)
0927 10:49:23.036012 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.036017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.036274 (+ 257us) server_negotiation.cc:407] Connection header received
0927 10:49:23.036514 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.036518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.036573 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.036629 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.037238 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49394: BlockingRecv error: recv got EOF from 127.0.0.1:49394 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 10:49:23.073714 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.072104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.10.96.253:43443)
0927 10:49:23.072262 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.072266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.072736 (+ 470us) server_negotiation.cc:407] Connection header received
0927 10:49:23.072865 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.072869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.072927 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.073020 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.073538 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49398: BlockingRecv error: recv got EOF from 127.0.0.1:49398 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 10:49:23.141216 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.139812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.10.96.253:43443)
0927 10:49:23.139995 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.140002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.140235 (+ 233us) server_negotiation.cc:407] Connection header received
0927 10:49:23.140330 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.140333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.140385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.140466 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.141048 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49400: BlockingRecv error: recv got EOF from 127.0.0.1:49400 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 10:49:23.272709 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.271452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.10.96.253:43443)
0927 10:49:23.271616 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.271621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.271794 (+ 173us) server_negotiation.cc:407] Connection header received
0927 10:49:23.271973 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.271976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.272037 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.272127 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.272596 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49408: BlockingRecv error: recv got EOF from 127.0.0.1:49408 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 10:49:23.532341 557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:23.531163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.10.96.253:43443)
0927 10:49:23.531326 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:23.531332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:23.531449 (+ 117us) server_negotiation.cc:407] Connection header received
0927 10:49:23.531640 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:23.531643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:23.531691 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:23.531783 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:23.532232 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35656: BlockingRecv error: recv got EOF from 127.0.0.1:35656 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 10:49:23.846570 650 catalog_manager.cc:1259] Loaded cluster ID: cf9328f936cf464fa78866256ad57c4a
I20260927 10:49:23.846659 650 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: loading cluster ID for follower catalog manager: success
I20260927 10:49:23.848296 650 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: acquiring CA information for follower catalog manager: success
I20260927 10:49:23.849402 650 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9068139b7004eb9ba286f82ccd73641: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:49:23.902791 688 catalog_manager.cc:1259] Loaded cluster ID: cf9328f936cf464fa78866256ad57c4a
I20260927 10:49:23.902906 688 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: loading cluster ID for follower catalog manager: success
I20260927 10:49:23.904593 688 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: acquiring CA information for follower catalog manager: success
I20260927 10:49:23.905855 688 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1830722bcec24bdc951ab28d08657e17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:49:24.051085 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:24.046747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.10.96.253:43443)
0927 10:49:24.046986 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:24.046990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:24.047054 (+ 64us) server_negotiation.cc:407] Connection header received
0927 10:49:24.047235 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:24.047238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:24.047367 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:24.047499 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:24.047785 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.048314 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.049180 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.049422 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.049687 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:24.050121 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:24.050595 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:24.050715 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:24.050871 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 10:49:24.050939 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260927 10:49:24.058732 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:24.054440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56772 (local address 127.10.96.252:34485)
0927 10:49:24.054719 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:24.054728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:24.054870 (+ 142us) server_negotiation.cc:407] Connection header received
0927 10:49:24.055106 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:24.055109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:24.055232 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:24.055369 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:24.055611 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.056161 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.057081 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.057334 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.057454 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:24.057822 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:24.058301 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:24.058455 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:24.058499 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 10:49:24.058562 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":99,"threads_started":1}
I20260927 10:49:24.066327 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:24.062198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.10.96.252:34485)
0927 10:49:24.062329 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:24.062333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:24.062523 (+ 190us) server_negotiation.cc:407] Connection header received
0927 10:49:24.062656 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:24.062659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:24.062724 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:24.062799 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:24.063169 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.063821 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.064989 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.065333 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.065434 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:24.065792 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:24.065884 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:24.065951 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:24.066113 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 10:49:24.066179 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:49:24.066700 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:24.062743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35664 (local address 127.10.96.253:43443)
0927 10:49:24.062903 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:24.062908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:24.062939 (+ 31us) server_negotiation.cc:407] Connection header received
0927 10:49:24.063193 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:24.063196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:24.063251 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:24.063296 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:24.063692 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.064421 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.065409 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.065775 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.065892 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:24.066194 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:24.066276 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:24.066321 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:24.066516 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 10:49:24.066581 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 10:49:24.067312 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:24.062184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.10.96.254:35271)
0927 10:49:24.062463 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:24.062467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:24.062856 (+ 389us) server_negotiation.cc:407] Connection header received
0927 10:49:24.063191 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:24.063195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:24.063370 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:24.063529 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:24.063770 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.064358 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.065333 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:24.065685 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:24.065801 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:24.066391 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:24.066806 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:24.066908 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:24.067089 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 10:49:24.067163 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20260927 10:49:24.070446 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 410
I20260927 10:49:24.077600 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 485
I20260927 10:49:24.083976 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 561
2026-09-27T10:49:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3123 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 10:49:24 dist-test-slave-wmg0 krb5kdc[779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 10:49:24 dist-test-slave-wmg0 krb5kdc[779](info): set up 2 sockets
Sep 27 10:49:24 dist-test-slave-wmg0 krb5kdc[779](info): commencing operation
krb5kdc: starting...
W20260927 10:49:25.121979 10627 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/test-user.keytab.
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T10:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T10:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
Entry for principal kudu/127.10.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab.
I20260927 10:49:25.240628 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36027
--webserver_interface=127.10.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:36027,127.10.96.253:40103,127.10.96.252:34635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 10:49:25.356253 792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:49:25.356597 792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:25.356665 792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:25.356743 792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:25.360776 792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:25.360872 792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:25.360912 792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:25.360947 792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:25.360985 792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:25.366375 792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:36027,127.10.96.253:40103,127.10.96.252:34635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.254.keytab
--principal=kudu/127.10.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.254:36027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:25.367477 792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:25.368544 792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:25.377170 792 server_base.cc:1034] running on GCE node
W20260927 10:49:25.377221 798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:25.377221 797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:25.377311 800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:25.377952 792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:25.378976 792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:25.380146 792 hybrid_clock.cc:648] HybridClock initialized: now 1790506165380133 us; error 25 us; skew 500 ppm
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:25.387866 792 init.cc:377] Logged in from keytab as kudu/127.10.96.254@KRBTEST.COM (short username kudu)
I20260927 10:49:25.388846 792 webserver.cc:460] Webserver started at http://127.10.96.254:34961/ using document root <none> and password file <none>
I20260927 10:49:25.389430 792 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:25.389599 792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:25.389832 792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:25.391917 792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data/instance:
uuid: "ccb16234aed24be5909073e9eeeee287"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "3ea50534c6901d32fc280d672fa2f8d3"
server_key_iv: "f6e7a9deea30f2e3e584eab288808b82"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.392513 792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal/instance:
uuid: "ccb16234aed24be5909073e9eeeee287"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "3ea50534c6901d32fc280d672fa2f8d3"
server_key_iv: "f6e7a9deea30f2e3e584eab288808b82"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.396342 792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 10:49:25.398895 807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:25.400187 792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 10:49:25.400328 792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal
uuid: "ccb16234aed24be5909073e9eeeee287"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "3ea50534c6901d32fc280d672fa2f8d3"
server_key_iv: "f6e7a9deea30f2e3e584eab288808b82"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.400441 792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:25.420533 792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:25.423718 792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:25.423942 792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:25.431107 792 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.254:36027
I20260927 10:49:25.431417 859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.254:36027 every 8 connection(s)
I20260927 10:49:25.432127 792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/data/info.pb
I20260927 10:49:25.434888 860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:25.437071 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 792
I20260927 10:49:25.437207 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-0/wal/instance
I20260927 10:49:25.437551 10627 external_mini_cluster.cc:1428] Setting key 148f2f1eecba3718d602274d0588d2f9
I20260927 10:49:25.438627 860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.253@KRBTEST.COM" created.
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:25.456697 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.440855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36027 (local address 127.0.0.1:49688)
0927 10:49:25.441429 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.441505 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.442323 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.442991 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.442996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.443457 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.443798 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.443815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.445522 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.445526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.446863 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.446885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.447926 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.447930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.448095 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.448785 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.448808 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.451039 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.453555 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.453568 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.453582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.453836 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.454250 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.454252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.454254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.454370 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.455220 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.455226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.455357 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.455514 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.455793 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"spinlock_wait_cycles":2944,"thread_start_us":162,"threads_started":1}
I20260927 10:49:25.456702 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.440855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.10.96.254:36027)
0927 10:49:25.441267 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.441285 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.441612 (+ 327us) server_negotiation.cc:407] Connection header received
0927 10:49:25.442630 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.442634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.442816 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.442997 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.443992 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.445370 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.447009 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.447689 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.447929 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.451206 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.451213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.451229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.451260 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.453430 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.453968 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.453977 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.453982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.454129 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.454496 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.454501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.454504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.455075 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.455226 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.455639 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.455912 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260927 10:49:25.459574 860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:25.460333 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.460081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45286)
0927 10:49:25.460205 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.460274 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 10:49:25.460570 808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
I20260927 10:49:25.461522 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.461259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45294)
0927 10:49:25.461354 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.461420 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 10:49:25.461794 860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:40103: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab.
Entry for principal kudu/127.10.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab.
I20260927 10:49:25.466269 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:40103
--webserver_interface=127.10.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab
--principal=kudu/127.10.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:36027,127.10.96.253:40103,127.10.96.252:34635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 10:49:25.511041 860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } attempt: 1
I20260927 10:49:25.511951 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.511618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45310)
0927 10:49:25.511801 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.511873 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 10:49:25.512687 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.512489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45320)
0927 10:49:25.512589 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.512633 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 10:49:25.512925 860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:40103: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
W20260927 10:49:25.579114 867 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:49:25.579453 867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:25.579525 867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:25.579607 867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 10:49:25.581091 860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } attempt: 2
I20260927 10:49:25.582093 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.581712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45334)
0927 10:49:25.581861 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.581972 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 10:49:25.582777 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.582591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45340)
0927 10:49:25.582679 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.582723 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 10:49:25.582937 860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.253:40103: Network error: Client connection negotiation failed: client connection to 127.10.96.253:40103: connect: Connection refused (error 111)
W20260927 10:49:25.584568 867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:25.584646 867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:25.584690 867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:25.584726 867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:25.584760 867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:25.590257 867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:36027,127.10.96.253:40103,127.10.96.252:34635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.253.keytab
--principal=kudu/127.10.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.253:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:25.591360 867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:25.592391 867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 10:49:25.600732 873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:25.600737 875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:25.600818 867 server_base.cc:1034] running on GCE node
W20260927 10:49:25.601060 872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:25.601646 867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:25.602720 867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:25.603875 867 hybrid_clock.cc:648] HybridClock initialized: now 1790506165603853 us; error 34 us; skew 500 ppm
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:25.606684 867 init.cc:377] Logged in from keytab as kudu/127.10.96.253@KRBTEST.COM (short username kudu)
I20260927 10:49:25.607537 867 webserver.cc:460] Webserver started at http://127.10.96.253:44635/ using document root <none> and password file <none>
I20260927 10:49:25.608042 867 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:25.608098 867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:25.608258 867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:25.610280 867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data/instance:
uuid: "7e8c3fc7358040d3afe8799b8a2c542b"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "643e1843c80d90f62f561d6a22076dd5"
server_key_iv: "c14e2e685cd895026949fba88fef5ff9"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.610775 867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal/instance:
uuid: "7e8c3fc7358040d3afe8799b8a2c542b"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "643e1843c80d90f62f561d6a22076dd5"
server_key_iv: "c14e2e685cd895026949fba88fef5ff9"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.614346 867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 10:49:25.616621 882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:25.618042 867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:25.618172 867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal
uuid: "7e8c3fc7358040d3afe8799b8a2c542b"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "643e1843c80d90f62f561d6a22076dd5"
server_key_iv: "c14e2e685cd895026949fba88fef5ff9"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.618275 867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:25.629101 867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:25.632360 867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:25.632560 867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:25.639565 867 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.253:40103
I20260927 10:49:25.639843 934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.253:40103 every 8 connection(s)
I20260927 10:49:25.640535 867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/data/info.pb
I20260927 10:49:25.642710 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 867
I20260927 10:49:25.642831 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-1/wal/instance
I20260927 10:49:25.643162 10627 external_mini_cluster.cc:1428] Setting key 4e143269e227badc057c3740082d47ff
I20260927 10:49:25.643591 935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:25.647713 935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.96.252@KRBTEST.COM" created.
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:25.661906 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.649679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.10.96.254:36027)
0927 10:49:25.649811 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.649815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.650825 (+ 1010us) server_negotiation.cc:407] Connection header received
0927 10:49:25.651754 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.651758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.651822 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.651918 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.653119 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.653742 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.654846 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.655237 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.655356 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.658116 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.658119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.658121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.658145 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.659590 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.660209 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.660211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.660213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.660286 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.660697 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.660700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.660701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.660904 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.660995 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.661659 (+ 664us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.661772 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 10:49:25.662796 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.650171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36027 (local address 127.0.0.1:49694)
0927 10:49:25.650675 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.650717 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.651558 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.652133 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.652140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.652467 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.652976 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.652990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.653860 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.653864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.654726 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.654734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.655393 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.655397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.655551 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.656161 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.656176 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.657945 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.659707 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.659714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.659730 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.660111 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.660379 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.660385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.660388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.660574 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.661026 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.661036 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.661355 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.661611 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.661940 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":170,"threads_started":1}
I20260927 10:49:25.664302 935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab.
Entry for principal kudu/127.10.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab.
I20260927 10:49:25.668970 10627 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
/tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:34635
--webserver_interface=127.10.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab
--principal=kudu/127.10.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.96.212:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.96.254:36027,127.10.96.253:40103,127.10.96.252:34635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:49:25.674103 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.664745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45342)
0927 10:49:25.664860 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.664875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.664952 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.665728 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.665734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.665939 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.666045 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.666051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.667073 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.667076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.667698 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.667706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.668386 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.668389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.668462 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.669122 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.669142 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.670641 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.672529 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.672534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.672536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.672721 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.673004 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.673007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.673009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.673084 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.673573 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.673576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.673662 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.673773 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.673899 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 10:49:25.674103 860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } attempt: 3
I20260927 10:49:25.674116 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.665016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.10.96.253:40103)
0927 10:49:25.665329 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.665338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.665356 (+ 18us) server_negotiation.cc:407] Connection header received
0927 10:49:25.665388 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.665392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.665597 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.665707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.666140 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.666948 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.667824 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.668253 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.668368 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.670771 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.670781 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.670796 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.670822 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.672386 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.672823 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.672829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.672831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.672906 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.673164 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.673168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.673170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.673414 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.673604 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.673851 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.673978 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":112,"threads_started":1}
I20260927 10:49:25.676611 935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:25.677371 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.677097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42604)
0927 10:49:25.677215 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.677295 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 10:49:25.677687 884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
I20260927 10:49:25.678779 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.678386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42618)
0927 10:49:25.678689 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.678729 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
W20260927 10:49:25.679069 935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34635: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:49:25.683899 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.674927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.10.96.253:40103)
0927 10:49:25.675026 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.675030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.675041 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:49:25.675069 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.675072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.675107 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.675177 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.675760 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.676427 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.677915 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.678215 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.678368 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.680396 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.680400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.680402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.680428 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.682038 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.682653 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.682656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.682659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.682755 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.683125 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.683129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.683130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.683346 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.683426 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.683697 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.683793 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 10:49:25.683949 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.674604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45352)
0927 10:49:25.674735 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.674750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.674830 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.675261 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.675266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.675478 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.675627 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.675635 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.676537 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.676541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.677768 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.677796 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.678372 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.678375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.678482 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.679044 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.679061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.680267 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.682165 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.682170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.682174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.682520 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.682872 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.682875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.682877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.683013 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.683432 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.683436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.683542 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.683651 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.683789 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 10:49:25.685074 860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:25.685803 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.685570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42626)
0927 10:49:25.685681 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.685735 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 10:49:25.686460 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.686209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42642)
0927 10:49:25.686340 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.686393 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 10:49:25.686589 860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34635: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
I20260927 10:49:25.717746 860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } attempt: 1
I20260927 10:49:25.718640 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.718294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42644)
0927 10:49:25.718477 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.718551 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 10:49:25.719340 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.719127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42646)
0927 10:49:25.719229 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.719279 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 10:49:25.719477 860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34635: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
I20260927 10:49:25.728361 935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } attempt: 1
I20260927 10:49:25.729188 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.728890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42658)
0927 10:49:25.729052 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.729113 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 10:49:25.729933 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.729749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42662)
0927 10:49:25.729840 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.729882 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 10:49:25.730113 935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34635: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
W20260927 10:49:25.785019 941 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 10:49:25.785354 941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 10:49:25.785414 941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 10:49:25.785516 941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 10:49:25.789589 941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 10:49:25.789660 941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 10:49:25.789696 941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 10:49:25.789732 941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 10:49:25.789767 941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 10:49:25.794622 860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } attempt: 2
I20260927 10:49:25.795444 941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.96.212:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.96.254:36027,127.10.96.253:40103,127.10.96.252:34635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/krb5kdc/kudu_127.10.96.252.keytab
--principal=kudu/127.10.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.96.252:34635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 10:02:35 UTC on ccycloud.quasar-keauur.root.comops.site
build id 4675880
I20260927 10:49:25.795563 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.795196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42668)
0927 10:49:25.795394 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.795476 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 10:49:25.796314 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.796092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42672)
0927 10:49:25.796197 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.796250 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 10:49:25.796526 860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34635: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
I20260927 10:49:25.796872 941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 10:49:25.797984 941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 10:49:25.798281 935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } attempt: 2
I20260927 10:49:25.799042 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.798750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42688)
0927 10:49:25.798916 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.798973 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 10:49:25.799785 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.799541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42694)
0927 10:49:25.799677 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.799730 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 10:49:25.799949 935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.96.252:34635: Network error: Client connection negotiation failed: client connection to 127.10.96.252:34635: connect: Connection refused (error 111)
W20260927 10:49:25.807350 950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:25.807359 948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 10:49:25.807372 947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 10:49:25.807760 941 server_base.cc:1034] running on GCE node
I20260927 10:49:25.808332 941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 10:49:25.809533 941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 10:49:25.810720 941 hybrid_clock.cc:648] HybridClock initialized: now 1790506165810704 us; error 33 us; skew 500 ppm
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 10:49:25.813555 941 init.cc:377] Logged in from keytab as kudu/127.10.96.252@KRBTEST.COM (short username kudu)
I20260927 10:49:25.814467 941 webserver.cc:460] Webserver started at http://127.10.96.252:42021/ using document root <none> and password file <none>
I20260927 10:49:25.815033 941 fs_manager.cc:359] Metadata directory not provided
I20260927 10:49:25.815152 941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 10:49:25.815374 941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 10:49:25.817453 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data/instance:
uuid: "d602a05a834640da9719fa1d832006be"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "47109eb8c1ff2f40bdfe66321fb4087a"
server_key_iv: "4f3a77c4a66fd71dd4950fd256ce5f73"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.818128 941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal/instance:
uuid: "d602a05a834640da9719fa1d832006be"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "47109eb8c1ff2f40bdfe66321fb4087a"
server_key_iv: "4f3a77c4a66fd71dd4950fd256ce5f73"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.822186 941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 10:49:25.824741 957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 10:49:25.826105 941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 10:49:25.826211 941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data,/tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal
uuid: "d602a05a834640da9719fa1d832006be"
format_stamp: "Formatted at 2026-09-27 10:49:25 on dist-test-slave-wmg0"
server_key: "47109eb8c1ff2f40bdfe66321fb4087a"
server_key_iv: "4f3a77c4a66fd71dd4950fd256ce5f73"
server_key_version: "encryptionkey@0"
I20260927 10:49:25.826303 941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 10:49:25.843726 941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 10:49:25.847250 941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 10:49:25.847415 941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 10:49:25.854429 941 rpc_server.cc:307] RPC server started. Bound to: 127.10.96.252:34635
I20260927 10:49:25.854717 1009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.96.252:34635 every 8 connection(s)
I20260927 10:49:25.855540 941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/data/info.pb
I20260927 10:49:25.858532 1010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 10:49:25.862383 1010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:25.865641 10627 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu as pid 941
I20260927 10:49:25.865778 10627 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKx0Sgr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790505884435945-10627-0/minicluster-data/master-2/wal/instance
I20260927 10:49:25.866078 10627 external_mini_cluster.cc:1428] Setting key 6d3ab492ebd5056a97d44c18359e2250
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
I20260927 10:49:25.878604 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.864441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.10.96.254:36027)
0927 10:49:25.864604 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.864608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.865619 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 10:49:25.866924 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.866928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.866992 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.867074 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.868602 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.869390 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.870908 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.871250 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.871356 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.874406 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.874411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.874414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.874462 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.876105 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.876706 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.876709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.876710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.876788 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.877176 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.877181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.877183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.877444 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.877595 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.878241 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.878381 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 10:49:25.879662 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.866829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.10.96.252:34635)
0927 10:49:25.867191 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.867202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.867342 (+ 140us) server_negotiation.cc:407] Connection header received
0927 10:49:25.867457 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.867462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.867626 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.867767 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.868649 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.869852 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.871008 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.871577 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.871661 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.874878 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.874886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.874903 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.874933 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.876877 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.877398 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.877403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.877404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.877526 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.877793 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.877795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.877797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.878061 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.878257 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.878484 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.878606 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":262,"thread_start_us":169,"threads_started":1}
I20260927 10:49:25.879664 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.864834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.254:36027 (local address 127.0.0.1:49698)
0927 10:49:25.865412 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.865490 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.866687 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.867383 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.867395 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.867921 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.868403 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.868425 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.869605 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.869608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.870751 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.870762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.871411 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.871414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.871599 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.872307 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.872335 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.874205 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.876215 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.876225 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.876243 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.876586 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.876877 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.876881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.876883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.877072 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.877593 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.877601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.877971 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.878158 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.878524 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":138,"threads_started":1}
I20260927 10:49:25.881068 1010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.254@KRBTEST.COM
I20260927 10:49:25.890487 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.881659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.253:40103 (local address 127.0.0.1:45360)
0927 10:49:25.881758 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.881774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.881866 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.882143 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.882145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.882335 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.882464 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.882473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.883619 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.883622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.884268 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.884277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.884796 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.884799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.884876 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.885404 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.885419 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.886831 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.888757 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.888761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.888763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.889026 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.889414 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.889417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.889418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.889568 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.890018 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.890024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.890130 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.890242 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.890355 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 10:49:25.890657 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.881760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.10.96.253:40103)
0927 10:49:25.881883 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.881887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.881902 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:49:25.881988 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.881991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.882047 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.882130 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.882686 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.883505 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.884359 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.884683 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.884779 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.886966 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.886970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.886972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.886997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.888584 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.889180 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.889185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.889187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.889287 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.889692 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.889695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.889696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.889912 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.889997 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.890329 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.890479 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 10:49:25.891080 935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } attempt: 3
I20260927 10:49:25.891481 1010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } has no permanent_uuid. Determining permanent_uuid...
I20260927 10:49:25.892280 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.882391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.10.96.254:36027)
0927 10:49:25.882516 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.882523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.882536 (+ 13us) server_negotiation.cc:407] Connection header received
0927 10:49:25.882730 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.882734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.882780 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.882840 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.883676 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.884182 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.885206 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.885566 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.885678 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.888305 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.888309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.888312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.888342 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.890092 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.890574 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.890580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.890583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.890678 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.891052 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.891055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.891056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.891289 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.891537 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.891989 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.892108 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 10:49:25.895726 860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } attempt: 3
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.253@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.252@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.96.254@KRBTEST.COM for kudu/127.10.96.252@KRBTEST.COM
Sep 27 10:49:25 dist-test-slave-wmg0 krb5kdc[779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790506165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.96.253@KRBTEST.COM
I20260927 10:49:25.902477 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.891500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42708)
0927 10:49:25.891748 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.891761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.891834 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.892322 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.892324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.892514 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.892609 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.892617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.893597 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.893600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.894273 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.894280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.894899 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.894903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.894999 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.895462 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.895473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.897686 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.900603 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.900608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.900611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.900881 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.901251 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.901257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.901262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.901350 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.901950 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.901953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.902058 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.902160 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.902300 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260927 10:49:25.902844 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.891927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42716)
0927 10:49:25.892029 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.892042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.892109 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.892677 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.892679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.892836 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.892918 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.892925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.894492 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.894495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.895257 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.895267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.895942 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.895945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.896014 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.896457 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.896471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.897807 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.901313 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.901316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.901318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.901627 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.901881 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.901884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.901886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.901959 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.902337 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.902340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.902426 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.902515 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.902642 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":20}
I20260927 10:49:25.903378 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.892026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.10.96.252:34635)
0927 10:49:25.892372 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.892383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.892395 (+ 12us) server_negotiation.cc:407] Connection header received
0927 10:49:25.892431 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.892433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.892472 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.892554 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.893031 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.894388 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.895368 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.895735 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.895840 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.897933 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.897936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.897938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.897960 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.901218 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.901722 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.901724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.901726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.901789 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.902043 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.902046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.902048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.902226 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.902304 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.903028 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.903179 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":6784,"thread_start_us":98,"threads_started":1}
I20260927 10:49:25.903527 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.891709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.10.96.252:34635)
0927 10:49:25.892131 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.892137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.892152 (+ 15us) server_negotiation.cc:407] Connection header received
0927 10:49:25.892185 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.892187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.892218 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.892304 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.892706 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.893440 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.894385 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.894775 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.895196 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.897809 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.897813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.897815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.897836 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.900474 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.901051 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.901057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.901062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.901144 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.901528 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.901531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.901533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.901810 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.901905 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.903201 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.903384 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":367,"spinlock_wait_cycles":768}
I20260927 10:49:25.905407 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.896078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.96.252:34635 (local address 127.0.0.1:42728)
0927 10:49:25.896188 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 10:49:25.896203 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 10:49:25.896275 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 10:49:25.896758 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 10:49:25.896761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 10:49:25.896967 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 10:49:25.897065 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.897071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.898027 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.898030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.898839 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 10:49:25.898849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.899455 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.899459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 10:49:25.899557 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.899948 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 10:49:25.899962 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 10:49:25.901170 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 10:49:25.903681 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.903685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.903689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.903974 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.904301 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 10:49:25.904304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 10:49:25.904307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 10:49:25.904413 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 10:49:25.904879 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 10:49:25.904885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 10:49:25.905010 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 10:49:25.905090 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 10:49:25.905228 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 10:49:25.906070 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.896290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.10.96.252:34635)
0927 10:49:25.896574 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.896580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.896591 (+ 11us) server_negotiation.cc:407] Connection header received
0927 10:49:25.896616 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.896618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.896648 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.896742 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.897172 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.897906 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.898967 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.899341 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.899454 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.901321 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.901325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.901328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.901354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.903551 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.904105 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.904108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.904111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.904191 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.904558 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.904562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.904563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.904758 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.905694 (+ 936us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.905766 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.905882 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":150,"threads_started":1}
I20260927 10:49:25.906390 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.893428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.10.96.253:40103)
0927 10:49:25.893571 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.893577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.893698 (+ 121us) server_negotiation.cc:407] Connection header received
0927 10:49:25.893789 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.893792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.893832 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.893912 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.894583 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.895040 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.898259 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.898638 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.898755 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.902598 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.902601 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.902605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.902637 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.904398 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.904956 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.904959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.904961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.905037 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.905337 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.905340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.905342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.905572 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.905779 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.906132 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.906247 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 10:49:25.907130 935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: Bootstrap starting.
I20260927 10:49:25.907265 1010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: Bootstrap starting.
I20260927 10:49:25.908494 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: Bootstrap starting.
I20260927 10:49:25.909994 1010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:25.909994 935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:25.910892 935 log.cc:824] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:25.910892 1010 log.cc:824] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:25.911504 860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: Neither blocks nor log segments found. Creating new log.
I20260927 10:49:25.912236 860 log.cc:824] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: Log is configured to *not* fsync() on all Append() calls
I20260927 10:49:25.913194 10627 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 10:49:25.914422 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: No bootstrap required, opened a new log
I20260927 10:49:25.915192 1010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: No bootstrap required, opened a new log
I20260927 10:49:25.915210 935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: No bootstrap required, opened a new log
I20260927 10:49:25.914295 834 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ccb16234aed24be5909073e9eeeee287"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49712
I20260927 10:49:25.917644 860 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.917855 860 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb16234aed24be5909073e9eeeee287, State: Initialized, Role: FOLLOWER
I20260927 10:49:25.918174 860 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.918728 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.918927 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:25.919117 935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.919293 935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8c3fc7358040d3afe8799b8a2c542b, State: Initialized, Role: FOLLOWER
I20260927 10:49:25.919580 935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.919641 860 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:25.919740 834 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 10:49:25.919718 1010 raft_consensus.cc:348] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.919862 834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:25.919922 1010 raft_consensus.cc:729] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d602a05a834640da9719fa1d832006be, State: Initialized, Role: FOLLOWER
I20260927 10:49:25.920284 1010 consensus_queue.cc:260] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.920817 935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:25.921104 834 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.921593 1010 sys_catalog.cc:564] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 10:49:25.922113 834 leader_election.cc:288] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [CANDIDATE]: Term 1 election: Requested vote from peers 7e8c3fc7358040d3afe8799b8a2c542b (127.10.96.253:40103), d602a05a834640da9719fa1d832006be (127.10.96.252:34635)
I20260927 10:49:25.923434 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccb16234aed24be5909073e9eeeee287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e8c3fc7358040d3afe8799b8a2c542b"
I20260927 10:49:25.923658 909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:25.924592 1024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.924989 1024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:25.925032 909 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb16234aed24be5909073e9eeeee287 in term 1.
W20260927 10:49:25.926038 813 tablet.cc:2369] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 10:49:25.926887 808 leader_election.cc:302] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e8c3fc7358040d3afe8799b8a2c542b, ccb16234aed24be5909073e9eeeee287; no voters:
I20260927 10:49:25.927134 1020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 10:49:25.927156 983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccb16234aed24be5909073e9eeeee287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d602a05a834640da9719fa1d832006be"
I20260927 10:49:25.927376 983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [term 0 FOLLOWER]: Advancing to term 1
I20260927 10:49:25.927524 1020 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [term 1 LEADER]: Becoming Leader. State: Replica: ccb16234aed24be5909073e9eeeee287, State: Running, Role: LEADER
I20260927 10:49:25.927881 1020 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } }
I20260927 10:49:25.925853 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.928701 983 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb16234aed24be5909073e9eeeee287 in term 1.
I20260927 10:49:25.928709 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: This master's current role is: FOLLOWER
W20260927 10:49:25.929404 1053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:25.929492 1053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:25.930325 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccb16234aed24be5909073e9eeeee287. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.930506 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:25.932209 941 master_runner.cc:263] No leader master found from master d602a05a834640da9719fa1d832006be
I20260927 10:49:25.932272 941 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:49:25.932956 792 master_runner.cc:263] No leader master found from master ccb16234aed24be5909073e9eeeee287
I20260927 10:49:25.933034 792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 10:49:25.934167 867 master_runner.cc:263] No leader master found from master 7e8c3fc7358040d3afe8799b8a2c542b
I20260927 10:49:25.934243 867 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 10:49:25.934458 1033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:25.934518 1033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:25.934597 1063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 10:49:25.935532 1063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 10:49:25.938149 1057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 10:49:25.938230 1057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 10:49:25.940426 983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [term 1 FOLLOWER]: Refusing update from remote peer ccb16234aed24be5909073e9eeeee287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:25.940501 908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [term 1 FOLLOWER]: Refusing update from remote peer ccb16234aed24be5909073e9eeeee287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 10:49:25.941129 1020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:25.941555 1020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 10:49:25.945155 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccb16234aed24be5909073e9eeeee287. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.945137 1024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccb16234aed24be5909073e9eeeee287. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.945267 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:25.945268 1024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:25.945611 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.935077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.10.96.253:40103)
0927 10:49:25.935193 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.935199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.935415 (+ 216us) server_negotiation.cc:407] Connection header received
0927 10:49:25.935587 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.935591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.935639 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.935690 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.936554 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.937288 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.938552 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.938871 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.938946 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.941112 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.941116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.941119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.941145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.942857 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.943371 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.943375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.943377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.943475 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.944235 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.944239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.944241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.944489 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.944596 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.945272 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.945406 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:49:25.947968 1067 mvcc.cc:205] Tried to move back new op lower bound from 7333913255685574656 to 7333913255644688384. Current Snapshot: MvccSnapshot[applied={T|T < 7333913255685574656}]
I20260927 10:49:25.948438 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.948562 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:25.949846 1021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.949975 1021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:25.950424 1069 mvcc.cc:205] Tried to move back new op lower bound from 7333913255685574656 to 7333913255644688384. Current Snapshot: MvccSnapshot[applied={T|T < 7333913255685574656}]
I20260927 10:49:25.962054 1024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.962212 1024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be [sys.catalog]: This master's current role is: FOLLOWER
I20260927 10:49:25.962622 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccb16234aed24be5909073e9eeeee287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb16234aed24be5909073e9eeeee287" member_type: VOTER last_known_addr { host: "127.10.96.254" port: 36027 } } peers { permanent_uuid: "7e8c3fc7358040d3afe8799b8a2c542b" member_type: VOTER last_known_addr { host: "127.10.96.253" port: 40103 } } peers { permanent_uuid: "d602a05a834640da9719fa1d832006be" member_type: VOTER last_known_addr { host: "127.10.96.252" port: 34635 } } }
I20260927 10:49:25.962769 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287 [sys.catalog]: This master's current role is: LEADER
I20260927 10:49:25.962875 1070 mvcc.cc:205] Tried to move back new op lower bound from 7333913255685574656 to 7333913255644688384. Current Snapshot: MvccSnapshot[applied={T|T < 7333913255685574656}]
I20260927 10:49:25.963346 1063 catalog_manager.cc:1347] Generated new cluster ID: db00a38805b941d8992d9e4fb7977bab
I20260927 10:49:25.963420 1063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 10:49:25.967337 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.935564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.10.96.254:36027)
0927 10:49:25.935676 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.935682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.946162 (+ 10480us) server_negotiation.cc:407] Connection header received
0927 10:49:25.946802 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.946805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.946858 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.946916 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.947906 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.948687 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.961566 (+ 12879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.961931 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.962011 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.963634 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.963638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.963641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.963673 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.965209 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.965740 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.965744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.965746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.965823 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.966097 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.966101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.966103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.966297 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.966410 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.967059 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.967176 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 10:49:25.974181 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:25.944044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.10.96.252:34635)
0927 10:49:25.944159 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:25.944163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:25.948316 (+ 4153us) server_negotiation.cc:407] Connection header received
0927 10:49:25.948515 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:25.948519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:25.948571 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:25.948626 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:25.950729 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.951459 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.952557 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:25.952894 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:25.952973 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:25.958002 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 10:49:25.958008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 10:49:25.958012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 10:49:25.958041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 10:49:25.959810 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.960884 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.960888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.960889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.960996 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 10:49:25.961316 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 10:49:25.961320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 10:49:25.961321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 10:49:25.961584 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 10:49:25.961702 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:25.973876 (+ 12174us) server_negotiation.cc:299] Negotiation successful
0927 10:49:25.974021 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 10:49:26.001549 1063 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 10:49:26.002864 1063 catalog_manager.cc:1504] Loading token signing keys...
I20260927 10:49:26.019394 1063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccb16234aed24be5909073e9eeeee287: Generated new TSK 0
I20260927 10:49:26.020371 1063 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 10:49:26.047987 863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.044639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.10.96.254:36027)
0927 10:49:26.044769 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.044774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.046812 (+ 2038us) server_negotiation.cc:407] Connection header received
0927 10:49:26.047030 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.047034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.047089 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.047144 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.047837 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49736: BlockingRecv error: recv got EOF from 127.0.0.1:49736 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 10:49:26.049409 1016 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.045404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.10.96.252:34635)
0927 10:49:26.045554 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.045558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.048507 (+ 2949us) server_negotiation.cc:407] Connection header received
0927 10:49:26.048683 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.048686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.048734 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.048785 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.049283 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42732: BlockingRecv error: recv got EOF from 127.0.0.1:42732 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 10:49:26.050964 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.046173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.10.96.253:40103)
0927 10:49:26.046297 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.046301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.049820 (+ 3519us) server_negotiation.cc:407] Connection header received
0927 10:49:26.050192 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.050196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.050250 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.050303 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.050843 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45392: BlockingRecv error: recv got EOF from 127.0.0.1:45392 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 10:49:26.055481 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.053928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.10.96.253:40103)
0927 10:49:26.054036 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.054041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.054469 (+ 428us) server_negotiation.cc:407] Connection header received
0927 10:49:26.054705 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.054708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.054790 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.054842 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.055357 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45398: BlockingRecv error: recv got EOF from 127.0.0.1:45398 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 10:49:26.060674 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.059218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.10.96.253:40103)
0927 10:49:26.059319 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.059323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.059785 (+ 462us) server_negotiation.cc:407] Connection header received
0927 10:49:26.059969 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.059972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.060025 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.060077 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.060567 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45402: BlockingRecv error: recv got EOF from 127.0.0.1:45402 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 10:49:26.066433 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.064897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.10.96.253:40103)
0927 10:49:26.065003 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.065008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.065535 (+ 527us) server_negotiation.cc:407] Connection header received
0927 10:49:26.065726 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.065730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.065780 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.065831 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.066326 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45410: BlockingRecv error: recv got EOF from 127.0.0.1:45410 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 10:49:26.074450 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.072739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.10.96.253:40103)
0927 10:49:26.072892 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.072897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.073419 (+ 522us) server_negotiation.cc:407] Connection header received
0927 10:49:26.073662 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.073666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.073720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.073773 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.074338 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45414: BlockingRecv error: recv got EOF from 127.0.0.1:45414 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 10:49:26.086844 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.085021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.10.96.253:40103)
0927 10:49:26.085165 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.085170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.085771 (+ 601us) server_negotiation.cc:407] Connection header received
0927 10:49:26.085988 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.085992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.086048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.086102 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.086711 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45418: BlockingRecv error: recv got EOF from 127.0.0.1:45418 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 10:49:26.107805 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.105978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.10.96.253:40103)
0927 10:49:26.106114 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.106119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.106720 (+ 601us) server_negotiation.cc:407] Connection header received
0927 10:49:26.106957 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.106962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.107022 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.107081 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.107686 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45428: BlockingRecv error: recv got EOF from 127.0.0.1:45428 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 10:49:26.143728 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.142395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.10.96.253:40103)
0927 10:49:26.142556 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.142563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.142790 (+ 227us) server_negotiation.cc:407] Connection header received
0927 10:49:26.142972 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.142975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.143031 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.143121 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.143602 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45432: BlockingRecv error: recv got EOF from 127.0.0.1:45432 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 10:49:26.211483 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.210183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.10.96.253:40103)
0927 10:49:26.210329 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.210336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.210514 (+ 178us) server_negotiation.cc:407] Connection header received
0927 10:49:26.210636 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.210639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.210690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.210776 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.211320 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45440: BlockingRecv error: recv got EOF from 127.0.0.1:45440 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 10:49:26.342756 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.341607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.10.96.253:40103)
0927 10:49:26.341765 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.341773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.341899 (+ 126us) server_negotiation.cc:407] Connection header received
0927 10:49:26.342023 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.342026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.342083 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.342160 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.342629 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45442: BlockingRecv error: recv got EOF from 127.0.0.1:45442 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 10:49:26.602213 940 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 10:49:26.600962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.10.96.253:40103)
0927 10:49:26.601133 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:26.601138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:26.601326 (+ 188us) server_negotiation.cc:407] Connection header received
0927 10:49:26.601484 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:26.601486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:26.601541 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:26.601624 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 10:49:26.602094 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45450: BlockingRecv error: recv got EOF from 127.0.0.1:45450 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 10:49:26.930843 1053 catalog_manager.cc:1259] Loaded cluster ID: db00a38805b941d8992d9e4fb7977bab
I20260927 10:49:26.930930 1053 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: loading cluster ID for follower catalog manager: success
I20260927 10:49:26.932641 1053 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: acquiring CA information for follower catalog manager: success
I20260927 10:49:26.934012 1053 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e8c3fc7358040d3afe8799b8a2c542b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:49:26.939312 1057 catalog_manager.cc:1259] Loaded cluster ID: db00a38805b941d8992d9e4fb7977bab
I20260927 10:49:26.939371 1057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: loading cluster ID for follower catalog manager: success
I20260927 10:49:26.940943 1057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: acquiring CA information for follower catalog manager: success
I20260927 10:49:26.942063 1057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d602a05a834640da9719fa1d832006be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 10:49:27.120427 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:27.116314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.10.96.253:40103)
0927 10:49:27.116569 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:27.116576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:27.116600 (+ 24us) server_negotiation.cc:407] Connection header received
0927 10:49:27.116782 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:27.116786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:27.116941 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:27.117065 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:27.117328 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.117890 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.118778 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.119022 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.119128 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:27.119460 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:27.119939 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:27.120082 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:27.120117 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 10:49:27.120180 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
I20260927 10:49:27.127635 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:27.123911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.10.96.252:34635)
0927 10:49:27.124131 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:27.124136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:27.124186 (+ 50us) server_negotiation.cc:407] Connection header received
0927 10:49:27.124388 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:27.124391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:27.124507 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:27.124629 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:27.124840 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.125348 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.126202 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.126434 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.126533 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:27.126829 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:27.127247 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:27.127391 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:27.127424 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 10:49:27.127478 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260927 10:49:27.134590 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:27.130691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.10.96.253:40103)
0927 10:49:27.130988 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:27.130996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:27.131291 (+ 295us) server_negotiation.cc:407] Connection header received
0927 10:49:27.131428 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:27.131431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:27.131484 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:27.131556 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:27.131777 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.132315 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.133314 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.133623 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.133718 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:27.134053 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:27.134135 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:27.134180 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:27.134362 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 10:49:27.134438 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260927 10:49:27.134793 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:27.131087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.10.96.252:34635)
0927 10:49:27.131194 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:27.131200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:27.131313 (+ 113us) server_negotiation.cc:407] Connection header received
0927 10:49:27.131509 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:27.131512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:27.131564 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:27.131628 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:27.131918 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.132431 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.133628 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.133951 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.134050 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:27.134306 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:27.134385 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:27.134448 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:27.134595 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 10:49:27.134647 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 10:49:27.136432 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 10:49:27.130701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.10.96.254:36027)
0927 10:49:27.130950 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 10:49:27.130955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 10:49:27.131379 (+ 424us) server_negotiation.cc:407] Connection header received
0927 10:49:27.131481 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 10:49:27.131484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 10:49:27.131652 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 10:49:27.131783 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 10:49:27.132708 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.133542 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.134465 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 10:49:27.135274 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 10:49:27.135394 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 10:49:27.135736 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 10:49:27.136117 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 10:49:27.136192 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 10:49:27.136276 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 10:49:27.136323 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260927 10:49:27.140225 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 792
I20260927 10:49:27.146948 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 867
I20260927 10:49:27.152942 10627 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKx0Sgr/build/debug/bin/kudu with pid 941
2026-09-27T10:49:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3068 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6192 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (282711 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST